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Introduction 



This report describes the design and implementation of the three interventions tested in Project 
Upgrade, one of four experiments conducted as paid of the Evaluation of Child Care Subsidy 
Strategies . 1 The evaluation was a multi-site, multi-year effort to determine whether and how different 
child care subsidy policies and procedures and quality improvement efforts help low-income parents 
obtain and hold onto jobs and improve outcomes for children. Study staff worked with states and 
communities across the country to identify significant issues and develop hypotheses about the use of 
child care subsidy funds that could be rigorously tested in a series of experiments. A guiding principle 
of the study was that state (or community) interests and preferences should play a large role in the 
choice of research topics and strategics. 

The funds that flow to states through the Child Care and Development Fund (CCDF), administered at 
the federal level by the Child Care Bureau, have two puiposes. The major portion of the funds 
provides subsidies for child care for children of low-income working parents whose eligibility is 
determined by states within broad federal guidelines. States must use at least four percent of CCDF 
funding, with the addition of state matching funds, to improve the quality of child care for all 
children. It was the expressed intention of the Child Care Bureau that the study generate a set of 
experiments that examined aspects of the use of both types of funds. 

While some states expressed interest in testing some alternative policies governing the use of direct 
service dollars, many more were concerned about the effectiveness of their current use of funds 
intended to improve child care quality. Ultimately, study staff working closely with state and local 
staff, implemented four experiments, two that tested alternative subsidy policies and two that tested 
approaches to the use of quality set-aside funds. Project Upgrade in Miami-Dade County fell into the 
latter group of experiments. The study was a test of three different language and literacy 
interventions, implemented in child care centers in Miami-Dade County that served low-income 
families, most of whom received a subsidy for child care. Two of the three interventions had 
significant impacts on children’s language and pre-literacy skills. 

Purpose and Organization of the Report 

The policy and research contexts for the study, as well as detailed descriptions of the design and 
implementation of the evaluation and presentation of findings on the impact of the interventions on 
teachers and children can be found in an earlier report . 2 



1 The Child Care Bureau and the Office for Planning, Research and Evaluation (OPRE) of the 
Administration for Children and Families within the US Department of Health and Human Services 
launched this study in 2001. The study was conducted by Abt Associates Inc, with its research partners 
MDRC and the National Center for Children in Poverty of Columbia University. 

2 Layzer, J.I., Layzer C.J., Goodson, B.D., and C. Price. 2009. Findings from an Experimental Test of Three 
Language/ Literacy Interventions in Child Care Centers in Miami-Dade County. Final Report. US 
Department of Health and Human Services Administration for Children and Families, Office of Planning, 
Research and Evaluation. Washington, DC. 
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This report focuses more narrowly on the design and implementation of the three language/literacy 
interventions for preschool children that were tested in child care centers in Miami-Dade County. 

Over the last several years, there has been growing interest in how interventions are actually 
implemented in real world settings. Understanding the extent to which the interventions as 
implemented met the specifications of their design is important in an experimental evaluation, both 
when the evaluation reports no effects of the intervention and when it does. There could be a number 
of reasons why an intervention is ineffective: its implementation might have been too weak to achieve 
the hypothesized results; it may not have differed sufficiently from the services the control group 
received; or the underlying logic of the intervention could be flawed. A careful examination of 
whether the interventions were implemented as intended, and of what was happening in the control 
group, would allow the researcher to eliminate the first two possibilities, leaving the third as the most 
likely explanation for failure. 

An implementation study is equally important when, as in Project Upgrade, the evaluation reports 
positive impacts. Policymakers and program operators need a detailed understanding of what was 
actually tested, what level of implementation of the intervention was needed to produce the desired 
impacts, and what factors affected the implementation. This report is intended to address these issues. 

The organization of the report is as follows: Chapter 1 provides a summary of the design and findings 
of the evaluation; Chapter 2 describes the design of the three interventions in terms of two 
components - the professional development model and the curriculum itself; Chapter 3 examines the 
implementation of the professional development model; Chapter 4 describes the implementation of 
the three curricula and examines the extent to which similar activities occurred in classrooms in the 
control group; and Chapter 5 discusses the findings and their implications. 

Chapter One: Summary of Evaluation Design and 
Findings 

Project Upgrade was a two-year experimental test of the effectiveness of three different language and 
literacy interventions, implemented in child care centers in Miami-Dade County that served children 
from low-income families. One hundred and sixty-four centers were randomly assigned to one of 
three interventions, each of which used a research-based curriculum, or to a control group that 
continued with its existing program. The curricula, while grounded in a common set of research 
findings, differed in intensity, breadth, pedagogic strategies and use of technology. In each center, one 
classroom that served four-year-old children was selected for the study. Teachers and aides assigned 
to the three treatment groups received initial and follow-up training as well as ongoing mentoring 
over a period of approximately 18 months, from Fall 2003 to Spring 2005. All classrooms in the 
study, whether treatment or control, received an initial package of literacy materials (paper, crayons, 
books, tape recorders, books on tape, etc.). To reduce staff turnover, teachers in all four groups who 
remained in centers received $500 in July, at the end of each year of the study. 

The hypotheses tested by the study stipulated two kinds of outcomes: teacher behavior and 
interactions with children, and aspects of the classroom environment that support children’s language 
and literacy development, measured through direct observation; and children’s language and pre- 
literacy skills, measured by their performance on a standardized assessment. Study staff conducted 
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classroom observations in Fall 2003, Spring 2004 and Spring 2005. Four-year-old children in the 
study classrooms were assessed in Spring 2005. 

Key findings are summarized below and in Exhibits 1.1 and 1.2. Impacts are described in terms of 
effect sizes. Effect sizes are standardized measures of the magnitude of treatment effects. For each 
outcome measure, the effect size is equal to the estimated impact of the treatment, divided by the 
control group standard deviation (a measure of the variation in scores within the group). The 
standardization makes possible a comparison of the size of treatment effects across studies and, 
within limits, across outcome measures. 3 For example, if the effect sizes of a treatment on outcome 
measures A and B are 0.50, and 0.25, respectively, then the size of the treatment impact on A is 
considered to be twice the size of the impact on B . 

Findings 

• The initial observations, conducted before the interventions, showed that, across all groups, 
teachers engaged in few of the behaviors and interactions that have been shown to support 
children’s development of language and literacy skills. 

• Within six months of training, in Spring 2004, all three language/literacy interventions 
produced significant impacts on teacher behaviors and interactions with children that 
supported their language and literacy development; by Spring 2005, these impacts were 
generally more pronounced, and there were significant impacts on the number of classroom 
activities that involved literacy, and on literacy resources in the classroom. 

• The interventions had significant positive impacts on teacher behavior. These impacts were 
generally stronger for teachers whose primary language was Spanish than for their English- 
speaking counterparts. 

• Two of the three interventions. Ready, Set, Leap and Breakthrough to Literacy , had 
significant impacts on all four measures of emergent literacy outcomes for children: 
definitional vocabulary; phonological awareness; knowledge and understanding of print and 
an overall index of early literacy. The impact of the two effective interventions was much 
greater for children in classrooms with Spanish-speaking teachers than for children in 
classrooms with English-speaking teachers. 

• The two interventions that had impacts on child outcomes brought children close to or above 
the national norms on three of the four outcomes. On the fourth, definitional vocabulary, 
although children in the two treatment groups had significantly higher scores, they still 
lagged considerably behind the national norms. The impacts represent between four and nine 
months of developmental growth, depending on the outcome. 

• The interventions resulted in a substantial increase in the time spent on language and literacy 
activities, both teacher-directed and child-initiated. This did not eliminate other important 
developmental activities. Rather, time spent on each of the other activities was reduced 
slightly. 



Comparisons across studies must be approached cautiously. Even if the same outcome measure is used, the 
comparison assumes that the two study samples have similar standard deviations. Comparison of effect 
sizes for very different outcome measures may be misleading. 
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• There was a small but significant relationship between teachers’ educational attainment and 
some aspects of their behavior with children before the interventions. The training and on- 
going mentoring provided as an integral paid of the interventions eliminated this relationship. 
That is, as a result of the training and mentoring, less-educated teachers looked remarkably 
similar to their better-educated counterparts in the extent to which they provided activities 
that supported literacy. Teachers’ educational qualifications did not modify the impacts of the 
interventions on child outcomes. 



Exhibit 1.1 



Key Impact Findings: Teacher Instructional Behaviors and Classroom Environment 





All Teachers 3 
(n = 104) 


Spanish-dominant 
Teachers 13 
(n = 49) 


English-dominant 
Teachers' 3 
(n = 55) 


Domain/Construct (measure) 


Effect size 


Effect size 


Effect size 


Teacher behavior (OMLIT, 2005) 

Support for Oral Language 


.61*** 


.63** 


.55* 


Support for Phonological Awareness 


.49** 


.43* 


.52* 


Support for Print Knowledge 


.74*** 


.90** 


.54* 


Support for Print Motivation 


.43** 


.59* 


ns 


Classroom literacy environment 
(OMLIT, 2005) 

Literacy Resources 


.28* 


ns 


ns 


Literacy Activities 


.80*** 


.80*** 


.77** 



d Outcomes shown are combined outcomes for all teachers in all three treatment groups; the reference group for the impact is 
all teachers in the control group (n = 53). 

b Outcomes shown are combined outcomes for all Spanish-dominant teachers in all three treatment groups; the reference 
group for the impact is Spanish-dominant teachers in the control group (n = 26). 

c Outcomes shown are combined outcomes for all English-dominant teachers in all three treatment groups; the reference 
group for the impact is English-dominant teachers in the control group (n = 27). 

*** = p<.001, ** = pc. 01, * = pc. 05 



Exhibit 1.2 

Key Impact Findings: Child Language and Emergent Literacy (TOPEL, Spring 2005)* 





Children in 


Children in 




Classrooms with 


Classrooms with 


All Children 


Spanish-dominant 

Teachers 


English-dominant 

Teachers 


(n = 674) 


(n = 332) 


(n = 342) 


Effect Size 


Effect Size 


Effect size 



Definitional Vocabulary 


.30*** 


.39** 


ns 


Phonological Awareness 


.39 *** 


.55 *** 


ns 


Print Knowledge 


.63*** 


.86 *** 


.41** 


Early Literacy Index 


.53 *** 


.72 *** 


.36** 



d Outcomes shown are combined outcomes for the two interventions that showed significant impacts (RSL! and BTL). 
Results for the two treatments were combined since they were very similar and to provide additional statistical power. 
Outcomes for the individual curricula are shown in the final report on the impacts of the interventions (Layzer et al., 2009, 
op. cit.). Reference groups for the three columns have n = 509, n = 281, and n = 228, respectively. 

*** = pc.001, ** = pc. 01. * = pc. 05 
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Chapter Two: Design of the Three Interventions 



As is true for many early childhood interventions tested in recent years, the three interventions tested 
in Project Upgrade had two components; a professional development component; and a curriculum 
component. For Project Upgrade, it is critical to understand both components. To meet the needs of 
the child care centers, teachers and children in Miami-Dade County, the curriculum developers 
designed professional development strategies that went considerably farther than the training 
customarily offered in support of their curricula. The impact findings summarized in the last chapter 
must be understood as a result of the interventions as they were designed for Project Upgrade, and 
cannot be generalized to the more usual implementation of these curricula, all of which arc 
commercially available. This chapter describes the professional development model designed for this 
study and the three curriculum models used. 

The Professional Development Model 

For all three interventions, a single professional development model was agreed-upon by the 
developers, in consultation with the Early Learning Coalition of Miami-Dade County. It reflected a 
shared understanding of the challenges posed by the child care system in Miami-Dade County, which 
include: a large number of small centers with director/owners who do not necessarily have an early 
childhood education background; many relatively untrained staff; and few classroom materials and 
resources to support literacy. The model had two important features: 

• a staffing plan with several layers of supervision; and 

• a training plan that featured three sequenced training sessions over an 18-month period, 
combined with ongoing mentoring and support over the entire period. 

Staffing Plan 

One hundred and ten centers were randomly assigned to one of the three interventions. In these 
centers, one four-year-old classroom was selected to participate in the study. 4 To assist in training 
and to provide ongoing support for teachers and other staff in these centers, the developers 
determined that each intervention would need two full-time mentors (a caseload of approximately 1 8 
centers per mentor). To oversee and support the two mentors, each developer would provide an on- 
site coach (a part-time position, to be filled by a full-time employee who would carry out this in 
addition to other functions for the developer). Finally, to oversee the implementation of all three 
curricula, and to act as a liaison between the developers’ staff, the mentors, the participating centers, 
the Early Learning Coalition (ELC) and the evaluation staff, the ELC would provide a full-time 
Project Manager. 

Qualifications for the mentor position included: an educational background in early childhood 
education (bachelor’s degree or higher); and experience in an early care and education setting. In 
addition, because so many of the teachers and other center staff spoke Spanish as a first language, or 
were monolingual in Spanish, each developer needed at least one of the two mentors to be bilingual. 



4 See the Final Impact Report for the protocol used to select classrooms. 
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Training and Support Plan 

An important feature of the training plan was the recognition that both mentors and providers would 
need training and ongoing support. Mentors who would visit classrooms and support curriculum 
implementation had to be trained in the curriculum and in the developer’s approach to mentoring (i.e., 
based on observation in the classroom and responsive to the needs of individual teachers); and 
classroom staff who would implement the curriculum had to be trained and supported . 5 In many 
ways, the approach to training and supporting mentors mirrored the approach to training and 
supporting classroom staff. In neither case was it assumed that a one-time training would be adequate 
by itself to ensure adherence to the desired approach. 

Training for mentors was designed to be a two-day training session, with one day devoted to the 
curriculum itself and the second to the mentoring process. Coaching and ongoing support and 
consultation for mentors would be provided by the on-site coach, as well as through weekly meetings 
of the curriculum team. 

For teachers (and other child care staff), the model called for three sequenced training sessions: the 
first, a two-day session, would introduce classroom staff to the curriculum and provide training in the 
initial activities. A follow-up session after two to three months would cover more complex material 
and activities. A second follow-up session, at the beginning of the second year would serve as a 
refresher training for staff remaining in their classrooms as well as a training for newly-hired 
classroom staff. To support implementation of the curricula, and to guide teachers’ practice, the plan 
called for mentors to conduct visits to each of the classrooms approximately once every two weeks, 
but never less than once a month. 

The Curriculum Models 

The three curricula tested were selected by the Early Learning Coalition after a systematic and 
comprehensive review of language/literacy curricula had been conducted by Abt Associates’ staff . 6 
To be considered for the study, a curriculum had to meet the following criteria: 

• Provides support for children’s language and early literacy; 

• Provides support for all four of the elements of language and early literacy that research has 
shown to be predictive of later reading success: oral language; phonological processing; print 
knowledge; and print motivation; 

• Is appropriate for and has been used with children whose first language is not English and 
with low-income populations; 

• Is supportive of children’s home culture and language; 

• Is appropriate for both three- and four-year-olds (since the ELC was interested in introducing 
a curriculum in three- year-old as well as four-year-old classrooms); 

• Flas some preliminary evidence of effectiveness; and 



5 Typically, the developers of all three curricula offer an initial two-day training in their curriculum, with no 
ongoing support or mentoring. 

6 The review was funded by the US Department of Education as part of a different contract. 



Project Upgrade Implementation Study 



6 





• Has a training plan that would allow the curriculum to be implemented by child care center 
staff. 

Six of the curricula reviewed appeared to meet the criteria and were recommended to the ELC. Staff 
at the ELC reviewed the curriculum descriptions and interviewed three developers. One curriculum 
was rejected after the interviews because the developer believed it was not suitable for staff who were 
either monolingual in Spanish or would be more effectively trained in Spanish. Finally, the ELC 
chose to test two nationally-known curricula. Ready, Set, Leap! (RSL!) and Breakthrough to Literacy 
(BTL). The ELC also selected Building Early Language and Literacy (B.E.L.L.), a curriculum not on 
the Abt list, which was developed by a local academic, Dr. Wendy Cheyney, and at that time 
implemented in kindergarten classrooms in Miami-Dade Public Schools. 

The three curricula selected differed in instructional approach, breadth of approach, materials 
provided, intensity and cost, but all three focused on the development of early literacy skills and 
knowledge. All three included take-home components (books and materials to be used by families 
with children at home) and tools that teachers could use to assess children’s progress in the 
curriculum. 

Ready, Set, Leap! (RSL!; LeapFrog SchoolHouse Inc.) is a comprehensive curriculum with activities 
throughout the day that include math and science. A multi-sensory curriculum, it builds oral language, 
phonological awareness, print awareness and a language-rich environment through whole-group, 
small group and individualized instruction, including the use of three different interactive technology 
tools. The curriculum offers teachers a choice of two different instructional manuals: one focusing on 
a literature-based approach, the other on a theme-based approach. For Project Upgrade, a theme- 
based approach was selected by the developer. 7 The 120 lessons are based on themes such as school, 
the child, home, neighborhood, animals, community helpers, nature, food and transportation. 

Teachers organize their activities with children around multiple thematically-grouped trade books 
over a one- or two-week period (before moving to another set of books). 

The RSL! curriculum suggests the material to be presented and how it should be presented. For each 
lesson, the teacher is given a theme, encouraged to read the appropriate Big Book or another book 
from the Read- Aloud library, integrate the LeapPad and other activities related to the theme and use 
both books and activities with the whole group and with small groups of children. Her teaching 
strategies should support reading comprehension, questioning, predicting, summarizing and problem 
solving and should help children reflect on what they have learned. 

The teacher is expected to encourage and model the use of RSL! technology, which uses separate, 
unrelated materials. The LeapPad™ is an electronic story book system that uses controlled-language 
stories to focus on discrete concepts (e.g., color, number, letter shapes and names, word meaning, 
directionality of print). The LeapDesk™ is a simplified keyboard with key cards and is used primarily 
to help children learn to identify upper and lower case letters by name, to learn the symbol and name 
for numbers, to begin to learn letter-sound correspondence and, ultimately, to practice writing and 
spelling (i.e., encoding). Both the LeapPad and the LeapDesk technology have Spanish-language 

7 The decision to select a single approach for all the RSL teachers had the advantage of simplifying the 
training and mentoring process, but was also seen as more accessible to teachers, who are accustomed to 
the idea of “themes” (e.g.. Thanksgiving, Easter, Christmas). The trainers and mentors moved them on to a 
richer set of themes, but the basic idea was the same. 
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activities for children. The LeapMat™ is another tool for learning letter-sound and letter-name 
combinations by combining sound and touch on a large plastic mat. Several LeapPads and LeapDesks 
arc located in an activity center and are meant to be used throughout the day, each day by each child. 

As part of the Home Connection, children take LeapPad story books and Little Take-Home Books 
home with them so that parents can share the story-reading and alphabet-learning experiences with 
their children. The LeapDesk incorporates an assessment function which evaluates the child’s 
knowledge of letter names, letter sounds, and phonemic awareness. 

Building Early Language and Literacy (B.E.L.L.; Neuhaus Education Center) is an add-on, pre- 
kindergarten literacy curriculum designed to promote children’s general language proficiency, 
phonological awareness, shared reading skills, and print awareness. It entails two daily 15-minute 
whole-group lessons. One of the daily lessons typically involves oral reading (the teacher reads aloud 
from a Big Book 8 or a poster and the children repeat the rhyme or story) and a discussion of the story. 
The curriculum uses controlled language books, which are intended to serve as a springboard for 
language development, rather than as the sole form of literature in the classroom. The second daily 
lesson focuses on phonemic awareness and knowledge of letters and typically involves attending to 
the sounds in words (for example, forming compound words, detecting rhymes, alliteration) and 
activities targeting print knowledge (directionality, identifying letters and known words in text). The 
curriculum contains 12 units covering 48 weeks of instruction. Materials provided for small group 
activities in activity centers after the whole group lesson include letter cards and books and listening 
center materials (books on tape). Teachers are encouraged to create their own materials. Small 
versions of the Big Books are sent home with children. A teacher-administered test of phonological 
awareness and print knowledge (PELI) is included as paid of the curriculum. 

Breakthrough to Literacy (BTL; Wright Group/McGraw-Hill) is a comprehensive curriculum with 
activities throughout the day that include math and science. The language and literacy curriculum is 
built around a series of weekly books with a focus on reading aloud and active discussion about the 
book. At the heart of the curriculum are five “daily essential practices”: listening to and discussing 
stories; reading; writing; individualized software instruction; and talking, reading and writing at 
home. 

The central strategy in the Breakthrough to Literacy curriculum is to build children’s oral language 
skills - including vocabulary, speaking, aural comprehension - and comprehension of text in an 
emergent literacy sense, that is, with the teacher scaffolding children’s access to text through shared 
reading. To this end, the key component of the BTL classroom is the shared weekly book, the Book 
of the Week (BOW). These books are grouped thematically, and most are created for purely didactic 
purposes (that is, they target early childhood basic concepts such as shapes, colors, size, and other 
labels such as the names of rooms in a house). Teachers read a BOW to the whole group daily, using 
the whole group setting to employ techniques of dialogic reading, such as asking questions aimed at 
developing vocabulary and concept knowledge, checking children’s comprehension, and building 
motivation by using questions to make connections between events or characters in the BOW and 
children’s experiences. These shared reading experiences and extensions are also to be conducted in 



A Big Book, for those unfamiliar with them, is a large version of a children's book that allows the teacher 
to point to words or details of an illustration while reading to a group of children. 
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small group settings, and vocabulary, oral language, and comprehension skills are emphasized in 
whole group and small group settings outside the shared reading activity. 

A second key component of the BTL curriculum is daily writing to support reading and 
comprehension. Teachers are expected to provide daily opportunities for children to engage in 
activities such as writing in a journal, writing about the BOW that was read (writing to a prompt, such 
as, “How did Buster help?” — writing related to a book about a service dog named Buster). Teachers 
support developmental writing by encouraging children to draw pictures (which the teacher captions) 
and then to try to write on their own. 

Each child is expected to spend 8-12 minutes a day on the computer. Computer software provides 
individualized self-paced literacy activities for children, also organized around the weekly book, that 
focus on phonological and print knowledge, with additional books for use once the child has 
completed the BOW activities. It is also worth noting that the sophisticated software uses male and 
female voices, regional accents and speakers of English as a second language to accustom children to 
a variety of tones and accents. 

BTL also provides materials to be sent home with the child, as well as a report function in the 
Individualized Software Instruction that enables the teacher to track children’s progress from session 
to session. The software assesses the child’s strengths and weaknesses and targets areas in which the 
child needs more practice, creating an individualized sequence for each child, without any effort on 
the part of the teacher. 

As noted above, all three curricula include materials to be sent home weekly with each child: RSL! 
provides interactive books to be read at home; B.E.L.L. provides small book versions of the Big 
Books read and discussed that week; and BTL provides copies of the week’s book to be read aloud at 
home. All three provide some materials in Spanish for children with the aim of motivating reading, 
regardless of the language. All three meet the Florida Preschool Language and Literacy Learning 
Standards; two of the three, RSL! and BTL, also meet the state standards for a comprehensive 
curriculum, since they include math and science concepts. 
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Exhibit 2-1 



Strategies Used to Address Four Critical Aspects of Language/Literacy Development 



Skill/Knowledge Area 



Curriculum 

B.E.L.L. 



Oral Language 



Read-aloud with trade books (rich 
natural language) 

o Post-reading discussions target: 
comprehension & vocabulary 
development activities 
o Frequency: Daily during whole- 
group/circle time; no set time limit 
Independent reading of short stories 
with controlled (reduced) language on 
LeapPad (electronic storybooks) 
Theme important — many suggested 
follow-up activities based on theme 
related to target book (i.e., whatever 
book teacher reads aloud to whole 
group that week). 



• Read-aloud with Wright Group books 
(controlled (reduced) language books) 
o Post-reading discussions target: 

comprehension & motivation 
o Frequency: 2-4 times/week, in 15- 
minute large (6+ children) group 
sessions 

• Theme important — many suggested 
follow-up activities based on theme 
related to target book (i.e., whatever 
book teacher reads aloud to whole 
group that week). 



• Read-aloud with Wright Group books 
(controlled (reduced) language 
books) — Book of the Week (one book 
read aloud every day in week); teacher 
could supplement with other books 

o Post-reading discussions target: 
vocabulary development (primary) 
and comprehension 
o Frequency: Daily 10-15-minute 

sessions in large (6+ children) group 

• Independent reading/virtual shared 
reading with same Book of the Week 
and other (Wright Group) books on 
computer (electronic storybooks) 

• Theme important— many suggested 
follow-up activities based on theme 
related to target book (i.e., whatever 
book teacher reads aloud to whole 
group that week (Book of the Week)). 



Print Knowledge • LeapDesk: this tool enables the child to • 

engage in self-correcting (that is, the 
tool provides feedback or the correct 
answer) activities aimed at learning 
letter shape (see, touch, trace, fit like 
puzzle); letter name (hear), letter sound 
(hear) 

• LeapMat: this tool has the same self- 
correcting aspect and features letter 
name, letter sound, and beginning 
spelling 

• LeapPad: electronic books on which 
children can control the pace of reading 
with a stylus 



Laminated charts with poems and • Computer (Individualized Software 

short controlled language stories (e.g., Instruction, ISI): letter identification 
in which target letters or sounds are (letter name), letter sound, concept of 

over-represented): Direct instruction in word, directionality of print 

letter identification (e.g, find the letter o Frequency: daily @ 8-11 min./child 

‘m’), directionality of print (e.g., left to • ISI also includes electronic storybook 
right, top to bottom), concept of word & in which program tracks (highlights) 
sentence, punctuation (e.g., first word print, highlights words, etc. 
in sentence starts with uppercase . Small group activities (letter ID, story 

letter, sentence ends with period), cards) related to the Book of the Week, 

o Frequency: 1-3 times/week in 15- including using the small Take-me- 

minute large (6+ children) group home books (small newsprint versions 

sessions of the Book of the Week that children 

could take home with them), identifying 
target letters or sight words in the book 
of the week, identifying words (e.g., 
underline each word, to show the 

understanding of concept of word). 
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Strategies Used to Address Four Critical Aspects of Language/Literacy Development 



Skill/Knowledge Area 



Phonological Sensitivity 



RSL! 



LeapPad: electronic books featuring 
beginning and ending sounds of words 
and rhyming. 

Teachers encouraged to include songs 
and rhymes in whole group activities 



Print Motivation 



Read-aloud (with trade books); 
discussion: focus on motivation — 
relating books to children’s experience 
or introducing toys for center time that 
are related to the theme 
LeapPad: electronic books featuring 
short stories, basic concepts (color, 
shape, position, number, etc.). 
Developer considers that trade books 
inherently increase motivation. 



Curriculum 

B.E.L.L. 



Manipulatives and games involving 
breaking compound words into 
component words, breaking sounds in 
words apart and blending them 
together; Direct instruction 
o Frequency: 1-3 times/week in 15- 
minute large (6+ children) group 
sessions 

Songs and rhymes included in 15- 
minute sessions and recommended for 
follow-up small group activities. 
Read-aloud (Wright Group books)+ 
follow-up activities (2-4 times/week in 
15-minute sessions) 

Developer uses Wright Group books for 
convenience; no philosophical 
commitment or reason not to use other 
books, but units are based on WG 
books 



ISI: beginning and ending sounds, 
rhyming and alliteration, breaking 
sounds of words apart and blending 
together; interactive software (program 
corrects child and provides more 
practice or provides more challenging 
tasks) 

o Frequency: daily @ 8-11 min./child 



Read-aloud (Wright Group books); 
follow-up activities that seem fun 
and/or relate to children’s experience 
Developer considers work with books 
that children are able to read 
independently (even if “reading” rather 
than actually decoding) is inherently 
motivating. 




Chapter 3: Implementation of the Professional 
Development Model 



In this chapter we describe how Early Learning Coalition (ELC) and developer staff and mentors 
implemented the professional development model described earlier. Information for this discussion 
came from a number of sources, including: 

• meetings with curriculum developers and ELC staff; 

• observation of training sessions; 

• document review (resumes, training agendas, manuals); 

• observation of mentoring visits; 

• monthly meetings with all Upgrade staff and partners; and 

• informal interviews and discussions with individual mentor staff. 

Preparing for Implementation 

Our discussions with the ELC assumed a two-year implementation of the selected curricula. To 
accomplish this, the decision to embark on the experiment needed to be made in the Spring of 2003. 

In reality, the decision was made at the end of July 2003, after extended negotiation about the number 
of interventions to be tested (a decision that was contingent on the funds that the ELC could dedicate 
to the purchase of the curricula and professional training that accompanied them) and an equally 
extended process of selecting the curricula. With a good deal of help from ELC staff and staff from 
the two central agencies that administered child care subsidies, the recruiting of centers for the study 
was accomplished in the two-month period following the selection of the curricula. Centers were 
recruited by the beginning of the Fall 2003. However, before classroom staff could be trained to 
deliver the curriculum, there were many steps to be taken, including the hiring of on-site 
coach/trainers by two of the three developers; 9 hiring, assignment, and training of mentors; the 
collection of baseline observation data in the study centers; and the collection of LAP-D child 
assessments by the ELC. Once these and other essential activities were completed, classroom staff 
could receive initial training in November/December 2003. This reduced the implementation period 
to 18 months (mid-December 2003 to mid-June 2005). 

In addition to the steps noted above, the curriculum developers identified two concerns after visiting 
centers that had been recruited to the study. The first concern was that many centers were poorly 
equipped to put in place a strong literacy curriculum. They lacked many of the basic materials that we 
would expect to see in an early childhood setting: e.g., art and writing materials; whiteboards; tape 
players and audiocassettes; and a variety of children’s books, among other things. The decision was 
made by the ELC, in consultation with members of the study team, to provide a package of basic 
literacy materials to all the four-year-old classrooms in the study, including those in the control group. 
These had to be selected, ordered and delivered before teachers were trained. 

The developers’ second concern arose from the large proportion of teachers (47%) who had indicated 
that they would prefer to be trained in Spanish. (Not all of these were monolingual; almost half of 



9 BTL already had a trainer on staff who moved to Miami for the study. 
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teachers whose first language was Spanish reported that they spoke a mixture of English and Spanish 
in the classroom.) Although all three developers had expressed their willingness and readiness to train 
Spanish-speaking teachers, it was clear that, without a training plan specifically tailored to meet the 
needs of this population, the interventions were likely to be unsuccessful. The development of these 
plans was helped by the fact that all three of the on-site coach/trainers were fluent in both English and 
Spanish, and Spanish was the first language for all three of them. 

The final step in preparation for implementation was to hire, assign, and train the mentors who would 
be responsible for providing ongoing support for classroom staff. By design, approximately equal 
numbers of centers participating in the study were assigned to each of the two central agencies, and 
random assignment to treatment and control groups was done within the two groups, so that each 
agency had approximately 1 8 centers in each of the intervention groups (and 27 centers in the control 
group). As part of their overall contract with the ELC to support quality improvement, negotiated in 
Summer 2003, each of the agencies was responsible for the salaries of three of the six mentors (two 
per intervention) who were hired. The interviewing and hiring of the mentors, however, was 
conducted by the ELC Project Manager. 

Two of the mentors hired had master’s degrees in education and had taught kindergarten, first and 
second grades in the Miami-Dade Public Schools. One mentor was working on her master’s degree at 
FIU while working in the school system. The three remaining mentors had undergraduate degrees in 
ECE and two had experience working with one of the central agencies and their centers on child 
assessment and quality improvement. Three of the six were bilingual. Once hired, the mentors were 
divided into two groups, bilingual/monolingual English and then randomly assigned to one of the 
curricula, so that each curriculum had one bilingual mentor who could work with teachers who were 
monolingual in Spanish. 

Training Mentors 

The mentor training, conducted by each developer and their on-site coach/trainer in late Fall 2003, 
was a two-day training that combined an introduction to the elements of each curriculum with a focus 
on the approach to mentoring that each developer prescribed. 10 There were many commonalities 
among the three interventions in their approach to mentoring. All three emphasized that the mentor 
would observe classroom staff, model activities and strategies, and continuously elicit feedback from 
teachers on aspects of the curriculum that were working better than others and on areas in which they 
needed more help. 

There were, however, differences in the amount of flexibility allowed in the mentoring process. Of 
the three interventions, B.E.L.L. was the most prescriptive, BTL the least restrictive. While the 
B.E.L.L. mentor training envisaged a consistent mentoring strategy across classroom staff with 
different backgrounds, both RSL! and, especially, BTL emphasized the need for flexibility to ensure 
that they met the needs of classroom staff with a wide range of education and experience. Trainers 
urged mentors to spend time initially observing teachers in the classroom and to use their 
observations to suggest and model activities that were feasible for an individual teacher and fit into 
whatever program or schedule she was currently using. The more individualized approach stressed by 



10 The mentors participated in the training sessions for classroom staff held later in the year, solidifying their 
understanding of the curriculum they were to support. 
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BTL recognized that not all classroom staff would be equally comfortable with the computer software 
or equally able to grasp more complex concepts, at least initially 

Both BTL and RSL! trainers presented material primarily through lecture and demonstration, while 
B.E.L.L. trainers spent less than half of the time in this way, using most of the remaining time to 
engage mentors in exercises and activities that they would be expected to model for teachers in the 
classroom. This is not to say that the BTL and RSL! training sessions were not interactive; rather they 
followed closely their mentoring model, modeling behavior and activities and then asking for 
feedback and suggestions. All three sets of trainers checked regularly to make sure concepts were 
understood and to elicit ideas from the mentors on how to work with teachers. All three training 
sessions were led by the developers and their coaches. 

Training Classroom Staff in the Curricula 

Although the major focus of the training and mentoring was the lead teacher in each classroom, all 
three developers made a point of including classroom aides and center directors in all the training 
sessions. This turned out to be an important feature of the training strategy and assured some measure 
of continuity when teachers left and had to be replaced and trained. 

Each curriculum developer conducted three major training sessions for classroom staff. The first of 
these was a two-day training session for classroom staff and directors in late November/early 
December of 2003. The second set of training sessions were two-day sessions held in late 
January/early February of the following year. This schedule allowed the trainers to introduce the 
curriculum in a sequence, with more complex material covered in the second session, after classroom 
staff had had a chance to absorb and begin to implement the initial material and associated activities. 
The third training session, held in August of 2004, was a one-day refresher training for staff 
remaining in their classrooms as well as a two-day training for newly-hired classroom staff. 

The training sessions represented a substantial effort on the paid of developers, with national staff at 
BTL and RSL! training sessions and the original authors at the BTL and B.E.L.L. sessions. In 
addition, training sessions were set up so that at least paid of the content was delivered in Spanish for 
teachers who had indicated a preference for being trained in Spanish. Developers used different 
strategies to accomplish this: one provided a translator, who sat at a table with the Spanish-dominant 
teachers; another had the site coordinator/trainer do a parallel translation of the whole training; the 
third had no whole-group training in Spanish, but had the bilingual site coordinator/trainer facilitate 
one of the training rotation stations. In addition, some teacher materials were provided in Spanish: 
one developer (B.E.L.L) translated all the training materials, the other two each provided translations 
of at least one type of teacher resource material (lesson extensions in one case (RSL!)), and the 
teacher’s guide in the other (BTL)). 

As with the training for mentors, very little of the training was didactic, although trainers for each 
curriculum began the initial session with an overview of the rationale underlying each element of the 
curriculum, including the research supporting it. Beyond that introduction, the sessions were highly 
interactive with many hands-on activities for trainees. In the judgment of study staff who observed 
the training sessions, all three were equally effective in engaging participants. 
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The training schedule conceals the reality that training was ongoing throughout the 18-month 
implementation period. The decision to train both directors and aides in addition to teachers was an 
astute one, because it ensured some continuity when teachers left their jobs. (Discussion of turnover is 
in the section below.) However, replacement teachers had to be trained and, to achieve this in a timely 
fashion, on-site trainers and mentors had to schedule individual training sessions. In addition, not all 
classroom staff were able to attend the scheduled training sessions, so that additional group training 
sessions had to be scheduled close to the planned session. No provision was made by the ELC to pay 
centers for substitute staff while classroom staff attended training sessions so that, while attendance 
was high at the initial session, some directors were reluctant or unable to release staff for subsequent 
training sessions. It seems likely that a stipend to cover the cost of substitute staff would have reduced 
the need for additional group training. However, the challenge of training new staff is probably 
irreducible. 

The Mentoring Process 

Each curriculum was assigned two mentors, paid for by the ELC, and supervised by the on-site 
coach/trainers employed by the developers. Each mentor was responsible for approximately 18 
classrooms, which she visited twice a month, on average. After the initial round of visits, the mentors, 
in consultation with their trainers, designed more flexible schedules that reflected the individual needs 
of classroom staff. Some classrooms required visits weekly or even more frequently, while others in 
which staff were moving quickly in implementing the curriculum required less frequent visits. More 
frequent visits were needed for teachers who were slower to grasp the concepts embedded in the 
curricula, or somewhat hostile to them, or who needed assistance in basic classroom management. 

The site coordinators also conducted mentoring visits, especially to new teachers or to teachers who 
were experiencing difficulty implementing the curriculum. The visits were similar across curriculum 
models, with each mentor visiting one or two classrooms in a morning, one or two classrooms in the 
afternoon, and completing paperwork at the end of the day. Each intervention used a systematic way 
of recording progress, rating fidelity of implementation, and providing instructional feedback to 
teachers. The forms used by the mentors reflect the developers’ ideas about key components of the 
curriculum and effective strategies to communicate them. While these forms do measure fidelity to 
the curriculum (discussed in the next chapter), their primary use was to guide the mentor’ s work with 
the teacher. They were used to identify specific areas for teachers to work on, such as conducting 
more activities in small groups, spending less time in whole-group activities, using graphic organizers 
to build vocabulary from the book of the week, and strategies for classroom management to help 
children focus. 

We do not show the fidelity checklists here, since like the training materials, they are proprietary. 
However, it is possible to describe the areas of observation and the ways in which mentors could add 
observational notes to the more formal assessment. 

For RSL!, at each visit, mentors were asked to rate as “observed” or “not observed”, the following 
aspects of the classroom environment and teacher behavior: 

• evidence of positive interactions between teachers and children and participation by all 
children (3 items) 

• aspects of the classroom environment that support children’s learning and specifically support 
literacy activities (7 items) 
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• evidence of implementation of the RSL! curriculum (25 items) 

The fidelity checklist for B.E.L.L. mentors had two components: the first noted implementation of 
aspects of oral language and shared reading; the second looked at phonological and print awareness. 

The first part of the checklist dealt with five topics: 

• the pre-reading discussion of the contents of the book and the teacher’s and children’s own 
stories (5 items); 

• the reading of the book by the teacher and retelling by children (4 items); 

• the rereading of the book, with discussion and open-ended questions (5 items); 

• reaction to the book in the form of related activities (5 items); and 

• print awareness (6 items). 

The second paid of the checklist covered four topics: 

• oral language development (4 topics); 

• phonological awareness , including word awareness, rhyming words, compound words, and 
alliteration (11 items); 

• print awareness and alphabet knowledge (5 items) 

• literacy support in other activities (4 items) 



The BTL fidelity checklist consisted of three components: the first covered “Essential Practices” 
(intended to be implemented daily); the second covered “additional daily practices” (specific features 
of classroom activities and organization), classroom environment and management; and the third was 
a very detailed report about computer settings, records of children’s individual performance and the 
overall performance levels of the class. 

The first part of the BTL fidelity checklist covered five “essential practices”: 

• BOW (Use of Book-of-the-Week; 14 items) 

• Daily Writing (14 items) 

• Small Group Activities (space for qualitative and descriptive information about small group 
activities) 

• Daily Computer (6 items per computer and 9 general — about computer settings and 
adjustments) 

• Home Connections (13 items) 



The second paid of the BTL fidelity checklist covered four “additional daily practices,” environment, 
and management: 

• Circle Time/Morning Meeting (10 items) 

• Centers (10 items) 
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• Environment (10 items) 

• Management (4 items) 



The computer checklist consisted of a detailed list of information from the computer-generated report 
of children’s activities, computer-based assessments, and so on. 

As we noted earlier, these measures were used to shape the support and technical assistance provided 
by the mentors and to develop individualized plans for each teacher to address areas of weakness. In 
addition, they served as an accountability tool for the on-site coaches, who could examine both the 
ratings and the plans developed from them and ensure that mentors were on-track and helping 
teachers to move forward. 

Challenges to Implementing the Professional Development Model 

By far - the most serious challenge for mentors and their coaches was teacher turnover. Teacher 
turnover is high in centers in Miami-Dade County — by some estimates as high as 45 percent per year - . 
The problem was only slightly ameliorated by the $500 retention stipends offered to all teachers. 

Over the period of the interventions, teacher turnover was 28% in RSL! classrooms, 42 % in BTL 
classrooms and 44% in B.E.L.L. classrooms. (In control classrooms, two-year - turnover was 49%. ) 
Most of the teacher turnover in B.E.L.L. classrooms occurred in the first year; in Year 2 of the study, 
turnover was only 5%. For the other two curricula and for the control group, turnover rates were 
roughly the same for each of the two years. As noted earlier, the developers made appropriate 
provision for training replacement teachers. Because aides, and in many instances center directors, 
had been trained on the curricula, they were able to provide guidance for new teachers and ensure 
some consistency during the transition. However, the need for on-going training (as opposed to 
mentoring) was greater than developers anticipated and made considerable demands on the time of 
the on-site coaches. 

In spite of the agreements signed with Abt and the ELC at the beginning of the study, there were 
some problems with lack of cooperation on the part of some center owners and administrative staff. 
Center directors were mostly supportive of the interventions and tried hard to ensure that teachers 
were able to attend the training sessions. However, in addition to the turnover in teachers, there was 
also turnover among center owners and owner/directors. In this situation, the ELC Project Manager 
and the coach had to spend time with the new director explaining the study and the intervention that 
was being implemented in their center. Usually, but not always, this effort was successful in 
continuing the center’ s participation, but there was usually less enthusiasm on the part of 
administrative staff and less willingness to release teachers for training sessions. Some directors 
prized their role as teacher supervisors and felt threatened by the mentor’s role as instructional guide. 
In these cases, the teachers often found themselves caught between the two, and mentors and coaches 
needed to meet repeatedly with directors to build trust and understanding. 

Mentors’ ability to shape teacher’s behavior and classroom practices was also hindered by lack of 
basic classroom management skills on the part of some of the teachers. While the mentors were 
ready and willing to deal with this problem, it absorbed a good deal of time, and meant that some 
teachers lagged behind in implementing the curriculum. In practical terms, what this meant was that 
some classrooms were visited weekly or even more frequently during the initial six months of the 
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intervention; others, where teachers were seen as moving steadily toward full implementation, were 
visited once a month. While it is almost certainly the case that any mentoring model would allow for 
this variation, coaches and mentors felt that, in the absence of an evaluation timetable, it would have 
been better to screen teachers and provide pre-intervention services to develop basic classroom 
management skills for those teachers who lacked them, before embarking on training them in the 
curriculum. This would entail only a slight modification of the professional development model. 

Chapter Four: Implementation of the Curriculum 

The final question about implementation deals with the extent to which, given the materials, training 
and ongoing mentoring they received, classroom staff were able to deliver the curriculum. This 
chapter describes our efforts to address this question, both during the course of the interventions, and 
after the study was completed. 

During the interventions, implementation of the curriculum for the purposes of the evaluation was 
examined in two different ways: 

• by mentors and coaches, using a global rating developed by Abt for the study and basing their 
rating on scores assigned on their own fidelity checklists, and 

• by senior evaluation staff, who conducted qualitative observations and interviewed mentors 
about implementation challenges and factors influencing successful implementation. 



After the study was completed, and because of our dissatisfaction with the global rating, we attempted 
a third strategy: 

• mapping the items in the three fidelity checklists onto items from the Observation Measures 
of Language and Literacy Instruction (OMLIT), 11 the battery of observation measures used by 
evaluation staff to collect data on the classroom environment, teacher behaviors and 
interactions with children related to support for language development and literacy skills. 



Below, we describe what we learned from each approach. 

Using Mentors and Coaches to Rate Fidelity of Implementation 

Probably the best way to assess the degree to which a curriculum is implemented as planned is for 
evaluation staff to work closely with the designers of the curriculum, before the intervention begins, 
to develop an observation measure which can then be used by objective and carefully-trained staff to 
carry out the assessment. This strategy has several advantages: it can ensure that all essential elements 
of the model are incorporated into a measure; and it can also be used to observe the extent to which 
participants in the control condition are exhibiting these same behaviors without the benefit of an 
intervention. By comparison, its disadvantages are minor. Because of resource constraints, it is 



1 1 Goodson, Layzer, Smith, and Rimdzius, 2004. Details about the process used in mapping items from the 
fidelity checklists to OMLIT variables are provided in Appendix A. More information about the OMLIT is 
also provided in Appendix B. 
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unlikely that these observations can be carried out repeatedly, to help us understand the dynamic 
process of implementation; it is more likely that they will be earned out at the end of the intervention 
(or at best, once again, part-way through the intervention). In addition, observers will need to receive 
some training to understand the curriculum, as well as training in applying the measure. 

For Project Upgrade, in addition to resource constraints, our decision about how to assess 
implementation was complicated immensely by the simultaneous testing of three very different 
curricula, using very different fidelity measures for the specific purpose of informing the mentoring 
process. Developing and applying a unique and objective measure for each of the three would have 
been time consuming and not possible within the time constraints of the study. Therefore, to assess 
the extent to which the interventions were implemented, study staff had to rely on assessments made 
by mentors and coach/trainers. In an effort to structure these assessments, we created a five-point 
summary rating scale to be used as a common metric by mentors and coaches in all three 
interventions. They were asked to base their rating of centers’ implementation on their scoring of 
their own detailed fidelity checklists (described above), condensing the scores to fit each point on the 
rating scale from 1 (not implemented) to 5 (fully implemented). 

The scale did not work well, especially for interim judgments, for reasons discussed below. Flowever, 
if we are willing to accept the final ratings made by the mentors, by the end of the second study year, 
all three curricula were judged to be fairly well or well-implemented in the majority of centers. 

Across the three interventions, a similar small number of centers (3 to 4) in each group were still not 
implementing the curricula at a satisfactory level. In some cases, this reflected a teacher, newly hired 
late in Year 2, who was not sufficiently familiar with the curriculum; in others, there was resistance 
on the part of the director or the teacher or both. 

There are many reasons why we should be wary of mentor judgments. First, we know from their own 
data that mentors for the same intervention, using their own measure, were differentially accurate in 
their assessment of implementation. The finer-grained the distinctions they were asked to make, the 
more likely they were to make an accurate assessment on individual items. Some mentors 
systematically made harsher judgments, others overlooked areas of weakness if they felt the teacher 
was making a good-faith effort. By itself, this effect could be countered if outside observers 
conducted periodic observations, using the developers’ own measures. This would necessitate careful 
training to ensure that the observers understood the curriculum, as well as the observation measure. 
Flowever, the final reason why mentors' judgments are flawed is that the judgments made for the 
rating, (i.e., partially implemented, fairly well-implemented, etc.) meant something different to the 
mentors than they might to external observers. They judged an individual teacher to be implementing 
well if she was implementing at the level they expected her to achieve at that point in time. Most of 
the mentors were former teachers and were accustomed to thinking in this way about their students. A 
third-grader may be tested and judged to be “proficient” in reading, but that does not imply he is a 
proficient reader. Rather it means he is at the stage in reading that we expect a third-grader to reach. 
These judgments are especially confusing when they are made early in the study, but even summative 
judgments at the ends of the study may be affected. Because their fidelity measures reflected their 
own effectiveness (as coaches) as well as the teacher’s compliance, there may have been some halo 
effect. 
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Qualitative Observations of Curriculum Implementation 



In addition to analyzing the mentors’ ratings of curriculum implementation, we were eager to 
understand and communicate what each of the curricula looked like when it was well-implemented. 
To accomplish this, in the Fall of 2004 and Spring of 2005, Abt staff accompanied both mentors for 
each of the interventions on mentoring visits and interviewed the mentors after the visits. (On three 
occasions, the ACF Project Officer was able to accompany study staff on these visits.) Staff visited 15 
classrooms, five for each of three curricula. 

Beginning in Fall 2004, the mentors and their coaches were allowed to select the classrooms to be 
visited, and we proposed that they select, in each case, classrooms where implementation of the 
curriculum was proceeding as planned. These observations were not meant to judge how well the 
curriculum was being implemented, since the mentors and trainers were much better judges of that, 
but to be able to describe, for each intervention, what a relatively -well implemented version would 
look like and what role the mentors played. Below, we provide three vignettes from the many that 
resulted from the visits, to illustrate each of the curricula. 



Ready, Set, Leap! In a visit to one classroom in early Fall 2004, children were engaged in writing activities 
in small groups. Some children were seated at tables writing first their own name and then a friend’s name. 
Others were using LeapFrog lighted writing surfaces to trace letters and words. In this classroom, the 
teacher conducted the class in Spanish for the most part, but struggled to conduct the letter-naming activity 
in English. The class was large - 22 children, — and no aide was present. Although this was only ten days 
into the program year, the four-year-olds, who all spoke Spanish as their first language, could identify all 26 
upper-and lower-case letters of the alphabet in English, and did so with great pride. The mentor explained 
that RSL! viewed this competence as an easy one to be mastered early in the program year so that children 
could move onto more challenging activities. 



I 



In another class observed, two children worked together on letter-sound identification at the Leap Desk, 
which was one of several activity centers in the classroom. In this classroom, all the RSL! technology was 
grouped in the one center and the teacher rotated children in small groups through the center. After the 
children had spent some time in the activity centers, the teacher gathered them for circle time and engaged 
them in a discussion of the picture book about Emperor penguins that she had earlier read aloud to them. 
In another classroom, no children were using the technology during the observation, but the teacher was 
working with a group of 10 children on letter-sound recognition, while the aide worked with the other half 
of the class on phonics (word families: -at and -un). Both groups were using teacher-made materials rather 
than materials provided by the developer. 
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B.E.L.L. In one classroom visited, the two teachers divided the class into two groups of ten children and 
conducted simultaneous sessions so that, while one group was engaged in shared reading, the other group 
was engaged in a phonological awareness/letter knowledge activity. The groups switched teachers and 
activities in the afternoon. In the shared reading group, the children listened to a story on tape while the 
teacher turned the pages of the book. At intervals, she paused the tape for comprehension checks, asking 
the children questions about the story. At the end, the teacher repeated the story, with the children 
saying the lines of the story out loud. In the other group, the teacher was playing “thumbs up/thumbs 
down” - children put their thumbs up when the teacher said a pair of rhyming words and down when the 
pair did not rhyme. They then explored an alliteration chart (poem) that focused on words that begin with 
the letter L. The teacher used this opportunity to review print concepts, asking questions such as “where 
do I start reading next, after I finish this line?” and “who can show me where a sentence begins?” The 
teacher also had children hold plastic snapping beads and push them together as they formed compound 
words, pulling them apart as they broke the words into their two separate components. 

In another class, working on the same activities in a different order, the mentor had identified two 
classroom management strategies for the teacher to work on: 

• When the children stand up or move around and interrupt an activity such as read-aloud, stop 

the reading and wait until they are all listening; 

• Collect the “break-it/make it” manipulatives when the activity is over so that the children can 

focus on their next activity. 

After the “make it/break it” activity (children held the plastic balls and broke them apart as they broke a 
word like “sunflower” into “sun’ and “flower” and pushed them together to make “haystack” from “hay” 
and “stack”), the teacher collected the plastic manipulatives and began working with the same alliteration 
chart focused on the letter L. The children were very engaged in this activity, so much so that several 
stood up and moved close to the chart, blocking it from view. The teacher ignored them and continued 
with the lesson. So she had used one of the mentor’s recommendations but not the other. 



Breakthrough to Literacy. In one classroom visited, there were several activity groups operating. In one, 
children were engaged in dramatic play. In the main part of the classroom, a small group was writing about 
one piece of the Book of the Week ( This Old Man/He Played Paddywhack), using a picture prompt from the 
book. Two children were copying a line of text from the book, one was writing about one aspect of the 
text (‘he gave the dog a bone”, one had added more detail to the picture and described the new scene (“it 
was a beautiful day”). At another table, children were molding play-doh into the shapes of items in the 
story (bones, a knee, a shoe, a thumb); at another table, children were counting out paper bones to match 
the number written on a card. At the sand table, one boy was digging up numbered bones, trying to find all 
20. On the rug, a girl was putting story cards in sequence. Two children were working at computers on 
different activities related to the BOW. The song “This Old Man” was playing on a CD player. The teacher 
sat with the writers while the aide circulated, talking to the play-doh group, the math group and checking 
on the computer users. After a few minutes, one of the children completed his time on the computer, 
moved his name-card and notified the child whose name-card was next in line. As the writers finished their 
work, they moved to a different activity center. 

In this classroom, the mentor had been working with the teacher for most of the year on shifting from 
whole-group to small-group and individual activities. During the whole of the observation, children spent 
the time in five or six small groups. However, the teacher still had some misgivings and wasn’t convinced 
that the small groups were effective. In this case, the teacher was following the mentor’s recommendation 
but still getting used to the feel of a different teaching approach. 
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Characteristics of Successful Implementers 



In interviews, mentors and coaches from all three curriculum models reported independently the same 
features of successful implementers: 

• a positive attitude towards instructional change. Mentors reported that perhaps the most 
important aspect of successful centers was willingness on the paid of both directors and 
classroom staff to refocus on specific literacy goals and adjust their space and routines to 
support those goals; 

• effective management of time. All of the mentors noted that the ability to organize lesson 
plans and manage time to accommodate all the planned activities were hallmarks of 
classrooms that could successfully implement a curriculum; 

• well-organized classroom space and effective classroom management. Once classroom 
materials have been set up and are available to children, the challenge is to cycle children 
through the different areas so that they are exposed to a variety of materials and activities 
over the course of a day and a week. High-implementing classrooms had effective strategies 
for handling transitions and helping children to take turns. For example, in one classroom, 
children placed cards with their names and a representative icon into a labeled pocket in the 
area where they chose to play. In another, children chose a song to play when it was time to 
move to another center. In many of the classrooms, at least initially, the mentors helped 
develop classroom schedules and provided lesson plans as a guide, intending to wean 
teachers from them as they became more competent in managing the classroom. In the end, 
however, most continued to provide lesson plans as way of maintaining a clear focus; 

• healthy working relationships among director, staff and parents. Implementation was 
smoother and more complete when directors trusted teachers to act as instructional leaders 
and make changes where necessary, with guidance from mentors. Directors’ involvement in 
classroom activities varied greatly; some taught regularly and visited the classroom often. 
Others were more concerned with running the center as a business and focused on those 
aspects of administration, visiting the classroom infrequently; and 

• frequent one-on-one interactions between teachers and children. In classrooms where the 
teacher was already practiced in making meaningful contacts with individual children, full 
implementation was more likely. In one classroom, the teacher used greeting time, as children 
gathered on the playground in the morning, to have brief conversations with each child. She 
treated these conversations as informal conferences to reinforce classroom routines and 
activities, help children plan their activities and encourage them individually. Conversations 
described actions, experiences and events and the teacher listened and responded to children’s 
comments and suggestions. Another teacher worked hard to maintain encouragement and 
one-on-one engagement with children at all ability levels, reinforcing the literacy goal 
positively whether answers given orally were correct or incorrect. In another classroom, some 
children were able to write names and addresses with or without adult help, while others 
made letter-like marks on the page. The teacher provided positive feedback to each child, 
regardless of ability level, reinforcing literacy goals as she did so. 
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Building Implementation Measures from Observation Data 



In neither of the approaches described above did we investigate what was happening in control group 
classrooms. We were aware that the control group centers continued to receive mentoring visits from 
Miami-Dade Public School staff as part of the subsidy agency’s efforts to improve the overall quality 
of centers serving subsidized and other low-income children. This constituted “business as usual” for 
those centers. 12 

In addition, we were concerned that the global ratings completed by mentors varied in their accuracy 
and also provided no insight into areas of weak vs. strong implementation. Therefore, after 
consultation with ACF staff, we elected to make a “post hoc ” effort to measure implementation of the 
three curricula. This effort, partially undertaken for BTL as paid of another study, 13 began with the 
developers’ own fidelity checklist (implementation measure). If the developer had not already done 
so, we grouped items conceptually (for example, phonemic awareness, print conventions). The next 
step was to identify items from the OMLIT battery of measures that reflected the underlying concept. 
The OMLIT is well-suited for this task, since the battery contains a very large number of individual 
items (a relatively small number of which were used to build the constructs for the teacher outcome 
measures). The categories in their fidelity checklists included some items that represented good 
developmental teaching practice, for which si mi lar items could be identified in the OMLIT. An 
example of such an item in BTL’s Reads and Discusses category is “teacher emphasizes 
vocabulary/oral language on 4-5 days of a typical week,” which could readily be linked to several 
items in the OMLIT. More challenging, but very important, were the categories that reflected the 
idiosyncratic approach of an individual curriculum. An example of such an item in B.E.L.L.’s Word 
Awareness category is “students push blocks for each word,” 14 which could not be linked to any item 
in the OMLIT. 

The task of the analyst, therefore, was to critically analyze the meaning of each item in the 
developers’ measures, identify one or more items from the OMLIT battery that reflected that 
meaning, and then compare the resulting set of OMLIT items with the developer’ s set of items to 
determine whether, taken together, the selected group of items was adequate to measure the category. 
The next piece of the task was to create an implementation index with items grouped conceptually. 
This was completed for both RSL! and BTL fidelity checklists. 

It proved impossible to complete the process for the B.E.L.L. fidelity checklist, however. While it 
was possible to identify OMLIT items that reflected some of the categories covered by the B.E.L.L. 
measure, some important aspects could not be measured. An example of a critical category for which 
we found almost no matching items in the OMLIT is Phonological Awareness, measured with 15 
items on the B.E.L.L. checklist. B.E.L.L. teachers were expected to spend a good deal of time on this 



12 None of the control centers was implementing a literacy curriculum, since this would have made them 
ineligible for the study. 

13 The Breakthrough to Literacy Fidelity Instrument: 2006-2007 was developed by Abt in collaboration with 
the BTL developers and instructional coaches in a study of Breakthrough to Literacy in the Chicago Public 
Schools funded by a grant from the U.S. Department of Education, Institute of Education Sciences (Layzer 
& Hurtig, 2006, unpublished document). 

14 This is really shorthand for “teacher engages students in an activity in which students push blocks for each 
word”. 
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category, supporting children’s phonological awareness development through physical expression - 
for example “feel syllables with hand under chin,” “students clap for each word,” “use connecting 
links or blocks to show awareness of syllables,” “use fists to ‘break it or make it.’” After lengthy 
review of what could and could not be measured with OMLIT items, we determined that the B.E.L.L. 
measure did not lend itself to this exercise, and did not proceed further. Our analyses of data from the 
2005 OMLIT were carried out for the newly created BTL and RSL! implementation indices only, 15 
although for each of them, we examined the extent to which B.E.L.L. teachers were engaging in 
similar practices. 

Translating Developers’ Fidelity Checklist Categories Into Implementation Index 
Dimensions 

BTL Implementation Index 

The BTL Implementation Index has seven conceptual dimensions, developed from the original BTL 
fidelity checklist’s five categories (Reads & Discusses; Writes; Individualized Software Instruction; 
Classroom Culture and Management Routines; and Reads, Writes, and Talks at Home), which 
correspond to the five “daily essential practices.” All except the category “Reads, Writes and Ta lk s at 
Home” for which there were no corresponding OMLIT items, are reflected in the Implementation 
Index as follows: 



BTL Fidelity Checklist Categories 


BTL Implementation Index Conceptual 
Dimensions 


Book of the Week (BOW): Reads & Discusses 


Comprehension while reading aloud: 

Comprehension activities conducted as part of 
shared reading experiences, when the teacher 
was reading aloud with more than one child. 


Comprehension in non-reading activities: 

Comprehension activities conducted outside of 
shared reading experiences. 


Vocabulary: 

Activities aimed at fostering vocabulary 
development. 


Oral language: 

Activities aimed at developing children’s oral 
language skills other than oral vocabulary. 


Daily Writing 


Writing: 

Activities intended to support the development of 
emergent writing skills 


Individualized Software Instruction (ISI) 


Individualized software instruction: 

Use of computer for instructional purposes. 


Classroom Culture and Management 
Routines 


Classroom culture and routines: 

Organization and management of the classroom 


Reads, Writes, and Talks at Home 


Not measured 

Use of take-home materials and activities with 
parents at home 



15 The measures, mapped onto the developers’ own measures, are attached in Appendix A. 
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The seven conceptual dimensions in the implementation index contain between one and fourteen 
items. (The specific items within each dimension arc shown in Exhibit 4.1.) Many activities were 
intended by the developer to be done in both large or whole group and in a small group setting or 
with individuals. 

RSL! Implementation Index 

The RSL! Implementation Index has three major dimensions, drawn from the original implementation 
fidelity checklist’s five categories: Interaction, Environment, RSL! Curriculum, Assessment, and 
Family Involvement. We were able to capture two of the three Interaction items, all seven 
Environment items, 1 1 of 25 Curriculum items, and none of the items from the Assessment and 
Family Involvement categories from the RSL! Fidelity checklist. Some examples of Curriculum 
checklist items that could not be matched to OMLIT valuables are: “RSL! materials are set up 
throughout the classroom,” “The teacher follows the daily schedule,” and “The teacher follows a 
lesson plan from the ‘Teacher’s Manual’ or the one provided by the project coordinator.” In addition, 
some aspects of items were not included because they were not measured, such as for “whole group 
language and literacy activities where each child is encouraged to participate ”: it is clear that the last 
part of this is probably the more important aspect of the item, but it was not something that we 
measured (see Appendix A-2). 



RSL! Fidelity Checklist Categories 


RSL! Implementation Index Dimensions 


Interaction 


Interaction: 

Teachers’ warmth and attentiveness to children, 
evidence of full-group activities. 


Environment 


Environment: 

Classroom organization and physical content 
including furnishings and materials for literacy 
instruction. 


Curriculum 


Curriculum: 

Use of “evidence-based” practices in early 
language and literacy instruction and use of 
specific curriculum materials (books) and tools 
(such as LeapPad, LeapDesk, and LeapMat). 


Assessment 


Not measured 

Use of assessment to guide instruction and 
completion of end-of-unit class assessment 
reports 


Family Involvement 


Not measured 

Use of information about children’s homes, 
interactions between home and school, and 
check-in/out system for parents to borrow RSL! 
materials 



RSLI’s curriculum strategy begins with creating a classroom that incorporates the features that 
researchers and practitioners have defined as “high-quality”, in terms of the interactions between 
teachers and children, and environmental aspects of the classroom. 

It is important to recognize, from the outset, that the measures developed as a result of the process 
described above, cannot be seen as providing a global implementation score or level, in the way that 
the developers’ own measures could do. As Exhibits A1 and A2 in Appendix A show, in each of the 
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two measures developed in this way, there arc a sufficient number of items that could not be linked to 
corresponding OMLIT items, that we need to be cautious in how they are used. We were interested, 
however, in examining the kinds of behaviors, activities and interactions that were more or less 
difficult for teachers to put into practice, to look at the extent to which control group teachers 
implemented any or all of these practices, and to look at commonalities across treatment groups, i.e., 
the extent to which teachers were implementing some practices, regardless of which curriculum they 
were trained to deliver. Of course the OMLIT, used as an interim outcome measure achieved some of 
these objectives. However, the creation of the implementation indices differed from the creation of 
OMLIT constructs in an important way. The constructs created for each implementation index 
represented the developer’s own hypothesis about effective practices. The OMLIT constructs were 
based on a research consensus, at the time they were created 16 . While there was considerable overlap, 
some interesting differences emerged, as the discussion later in the report suggests. 

Findings 

The data used for the analysis were collected in late Spring 2005, by independent observers using the 
OMLIT battery. Our description of the findings is couched in terms of the proportion of teachers in 
each group who are rated as partially or fully implementing the items that make up each of the 
curriculum dimensions. 

We present the findings for each of the two curricula separately, first discussing the results for the 
teachers and classrooms in the appropriate treatment group and then comparing them with teachers in 
the control group classrooms and with teachers who were trained in one of the other two curricula. 
After the overall findings, we present findings on implementation of the two curricula by the two 
major subgroups of teachers: those for whom Spanish was the primary language and those for whom 
English was the primary language. 

Implementation of BTL Essential Practices 

Overall, although large proportions of the teachers implemented the BTL curriculum strategics and 
activities, most implementation was only partial. For example, only one or two teachers were 
observed using small group activities, which meant that all of the others were missing a key aspect of 
the curriculum, which is differentiation of instruction to focus on individual children’s strengths and 
weaknesses. 17 While varying proportions of teachers in all groups implemented some aspects of BTL 
practices, more of the BTL group teachers than teachers in other groups implemented BTL strategies 
and activities. 

Teachers’ use of comprehension strategies while reading aloud. All BTL teachers were 
observed reading a book aloud to children, most (7 1 %) used a book that was related to a class theme 
(a strategy for bolstering comprehension), and almost all (89%) used one or more comprehension 
strategies during the read-aloud. About two-thirds of the teachers extended the reading with a 



16 For information on the psychometric properties of the OMLIT measures, see Judkins, D., St. Pierre, R., 
Gutman, G., Goodson, B., von Glatz, A., Hamilton, J., Webber, A., Troppe. P., and T. Rimdzius (2008). A 
study of classroom literacy interventions and outcomes in Even Start. Washington, D.C: US Department of 
Education, Institute of Education Sciences. 

17 The BTL mentors identified the same weakness in their own fidelity ratings. 
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discussion or other activity aimed at deepening comprehension; 18 most of those teachers used book- 
related questions to prompt discussion about the book. A much smaller proportion (3 1 %) of teachers 
made connections between the book and children’s experiences (to make the content less abstract or 
to enhance motivation) or explicitly connected the book to the classroom theme (to support the 
transfer or generalization of book content). (Exhibit 4.1) 

Teachers’ use of comprehension strategies in non-reading activities. Less than 30 percent of 
BTL teachers used comprehension strategies outside shared reading activities, and those who did 
conducted these activities in large groups rather than with individuals or in small groups, while the 
developer recommends using these strategies in large and small groups. Less than one-fifth (17%) of 
BTL teachers conducted discussions with children that were rich, abstract, or open-ended, or 
extended beyond the here-and-now, or related to children’s experiences (Exhibit 4.1). 

Teachers’ use of activities aimed at supporting vocabulary development. On this dimension, 
most teachers did none of the kinds of activities (such as calling attention to new words during a 
shared reading activity, highlighting the meaning of new words in discussions or conversations) 
intended by the developer. Only 20 percent of BTL teachers emphasized vocabulary during a read- 
aloud activity, for a smaller proportion (14 %) the quality of story-related vocabulary instruction was 
moderate to high, and a similarly small proportion worked to support vocabulary development outside 
the read-aloud activity. This area of early childhood development is one of the most challenging for 
teachers to support, particularly for teachers who themselves have more limited vocabulary or feel 
less confident about techniques for helping children learn new words, but vocabulary development is 
a critically important area for supporting children’s later reading comprehension(Exhibit 4.1). The 
weaker performance by teachers in activities to support vocabulary development is reflected in the 
lesser (though still significant) impact of the interventions on children’s vocabulary development. 

Teachers’ use of strategies to support oral language development. Most (77%) teachers 
provided opportunities for children to speak in a large group or whole group setting, and the quality 
of those opportunities in most was moderate to high — that is, the teacher extended discussions with 
multiple back-and-forth exchanges with individual students. In addition, more than half of the 
teachers also used dialogic reading techniques that involved children’s oral participation — either 
through asking higher-order questions or eliciting elaborated response. As in other activities, 
however, all of these opportunities occurred in large or whole groups rather than small groups or with 
individuals, which would be more developmentally appropriate and afford in-depth interaction with 
more children. 

Teachers’ use of strategies and activities to support the development of early writing. Almost 
three-quarters (71%) of the teachers provided at least two high-quality writing activities (that is, not 
just copying or de-contextualized writing, but rather writing for a puipose or within a familial - topic), 
a slightly higher proportion (77%) worked with all or most of the children in writing activities during 
the day and many (66%) managed to work with individual children or small groups of children on 
writing. In line with the developers’ intentions, almost 80 percent of teachers had children write on 
their own and did not insist on conventional spelling or letter formation, and more than two-thirds of 
classrooms had children’s own writing (more than just their names) on display. Overall, a majority of 



18 This rating captured only discussions or activities rated as moderate to high in quality — that is, lasting at 
least 5 minutes and extending or reinforcing the comprehension of the book. 
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teachers (66%- 83%) did more than 50 percent of what developers intended on this dimension. 

Writing activities that smaller proportions of teachers implemented include: integrating writing into 
other activities (43%); establishing a separate writing area with writing materials accessible to 
children (46%); and connecting writing to a class theme (14%) (Exhibit 4.1). 

Classroom use of individualized software instruction (computers). We were not able to 
document how the computers were used, only whether children used them during the observation. If 
children were using them daily as suggested, for the intended amount of time, we would have 
expected to observe children using them during at least 2 hours of the 2.5-hour observation period. In 
fact, we found that in 57 percent of classrooms, teachers had implemented this for the amount of time 
intended — at least as far as we could measure. It is possible that children also spent more time on the 
computer after the observation time, since children were in the classroom all day, and it is also 
possible that, in the 43 percent of classrooms where we did not observe children using the computer 
for the recommended amount of time, children used the computers when we were not observing, such 
as in the afternoon. 

Classroom culture and routines. All of the classrooms were well-organized and afforded children a 
variety of choices for independent activities. Aside from things that are common across preschool 
classrooms (such as having activity centers), one noticeable feature of these classrooms was that, in a 
large proportion of them (83%), children spent less than one-third of the time in management, 
routines or transitions. In almost one-third of the classrooms, children spent more than half of their 
time in age-appropriate “high-value” activities (reading, alphabet, oral language, sounds, writing, 
science/nature, math, creative play, dramatic play, block play, and fine motor play. In more than half 
of the classrooms (57%), children spent at least 20 percent of their time, outside the “routines and 
transition” portion, in a group of five or fewer children. This is not inconsistent with earlier 
statements; children spent a fair amount of time in activity centers and other small group activities, 
but teachers did not often use this time as an opportunity to work with them in the smaller group. 

How Did The Other Classrooms In Project Upgrade Compare With BTL Classrooms On The BTL 
Implementation Index? 

Teachers’ use of comprehension strategies while reading aloud. While the majority of teachers 
(71%) in the BTL group used books connected to a classroom theme during read-aloud activities, 
many fewer teachers in the control, RSL! and B.E.L.L. groups used books in this way (23%, 44%, 
and 12%, respectively). Although a majority of teachers in all three treatment groups used a variety of 
strategies designed to promote comprehension, BTL teachers were more likely to do so than teachers 
in the other two groups (89% vs. 78% and 76% for RSL! and B.E.L.L. respectively) and much more 
likely to do so than teachers in the con tied group (60%). Post-reading book-related discussions and 
activities in BTL classrooms were also notably more prevalent and of higher quality than in other 
groups. RSL! teachers were rated more highly on one item: almost twice as many related the book 
they were reading to the child’s experience as teachers in the other three groups (58% vs. 31%, 33%, 
and 30% for BTL, B.E.L.L., and control group teachers, respectively (Exhibit 4.1). 

Teachers’ use of comprehension strategies in non-reading activities. Although comprehension 
strategies outside shared reading experiences were not the most widely implemented strategy in BTL 
classrooms (less than one-third of BTL teachers used this strategy), they were very rarely used by 
control group teachers (6%), and only somewhat more frequently observed for teachers in the other 
two groups (19% and 12% for RSL! and B.E.L.L., respectively). 
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Teachers’ use of activities aimed at supporting vocabulary development. As we noted earlier, 
this was a weak area for BTL teachers. It proved to be similarly weak for teachers in the B.E.L.L. and 
control groups. Interestingly, more RSL! teachers used strategics to support vocabulary 
development — slightly more than other groups in shared reading but far more than all other groups 
outside shared reading activities (36% of RSL! teachers, as opposed to between 6 and 14% of 
teachers in other groups) . This latter kind of activity (using language games, songs, rhymes, 
explanations, discussions and other activities to encourage developing and expanding vocabulary) 
was rarely seen in B.E.L.L. and control group classrooms (6% and 9%, respectively). 

Teachers’ use of strategies to support oral language development. Like BTL teachers, about 
four-fifths of RSL! and B.E.L.L. teachers (81% and 79%) provided opportunities for children to 
practice communication skills, as did more than two-thirds of teachers in the control group. Between 
60 percent and 67 percent of teachers in the three treatment groups provided activities that combine 
communication and listening skills, compared with only 40 percent of teachers in the control group. 
The quality of oral language activities was higher in the BTL group of teachers than in the other three 
groups, although the proportion of teachers using high-quality oral language strategies was higher in 
all three treatment groups than in the control group. 

Teachers’ use of strategies and activities to support the development of early writing. Both 
BTL and RSL! emphasized the importance of children’s writing and similarly high proportions of 
teachers in both these groups engaged children in writing activities, encouraged children to write and 
displayed their writing. Much smaller proportions of B.E.L.L. and control group teachers engaged 
children in this way. In over three-fourths of BTL classrooms, teachers encouraged children to write 
on their own and supported developmental writing, while in Control group and B.E.L.L. classrooms 
the proportions ranged from just over half to less than two-thirds following this approach. 

BTL classrooms included more writing activities, and more high-quality writing activities than 
classrooms in other groups, with the difference being less pronounced between BTL and RSL! (but 
more pronounced between teachers in those two groups and teachers in the B.E.L.L. and Control 
group classrooms). Many more teachers in BTL classrooms than in the others worked with most or all 
students in writing activities throughout the day; again, the difference between BTL and RSL! 
classrooms was less pronounced. 

Classroom use of individualized software instruction (computers). Although implementation of 
the computer component was less than what developers might have hoped in BTL classrooms, with 
children in just over half of classrooms observed using computers for the recommended amount of 
time, there was dramatically less use of computers in classrooms in all of the other groups. Teachers 
in B.E.L.L. classrooms were least likely to use computers in the classroom (6% vs. 11% for RSL! and 
13% for Control group teachers). This item is the least reliable item on the index, however, as our 
observation period spanned a relatively small proportion of the whole program day, and it is possible 
that children were involved in activities on computers during other times of the day in classrooms in 
all groups, as long as computers were available. 

Classroom culture and routines. There were only two notable differences in classroom 
management across groups, with one important difference being that compared with all other groups, 
children in a larger proportion of RSL! classrooms spent less than one-third of their time in routines. 
The other big difference was that a larger percentage of Control group classrooms (nearly 74%) had 
children spending over 20% of their time in groups of five or fewer children. One possible 



Project Upgrade Implementation Study 



29 





explanation for this is that, in the control group classrooms, children were free to move to activity 
centers after a brief Circle Time activity. Teachers might observe those activities but not actively 
work with children in the small groups. Although, in substantial proportions of the treatment 
classrooms, children spent time in small group activities, a larger proportion of time was spent on 
large-group language and literacy activities than in the control group classrooms. 

Implementation Of RSL! Essential Practices 

Overall, large proportions of RSL! teachers demonstrated evidence of the characteristics of 
Interaction, and of implementation of the strategies and activities in the Environment dimension, but 
far smaller proportions (fewer than half of the teachers) demonstrated evidence of implementation on 
most (seven out of eleven) of the Curriculum dimension items. 

Interaction. The two items rated in this dimension looked at grouping and at teacher’s tone and 
attitude towards children. All RSL! teachers were observed to engage in some activities with the 
whole group. In fact, since this is the norm for prekindergarten classrooms, all classrooms in all 
groups in the study were observed to use whole-group activities, and since we did not qualify as to 
whether or not each child was encouraged to participate, this item does not capture key differences 
that might have been present. As far as using a positive tone and showing respect for children, over 
half of the RSL! teachers (58%) were observed to behave in this way. 

Environment. This scale measures generally accepted aspects of the physical environment of the 
preschool classroom, such as size, condition, and organization of furnishings; presence of activity 
centers, including a cozy reading area; materials representing diverse cultures and languages; 
materials to support emergent writing and display of children’s writing. All classrooms in all study 
groups were arranged with activity centers for children, nearly all RSL! classrooms (and all 
classrooms in all groups) had word cards for children to copy. Most RSL! classrooms had appropriate 
furnishings, and more than 60 percent had dedicated reading areas. Over half of RSL! classrooms had 
books representing diversity of culture and language, and a similar proportion had a variety of tools 
and materials for writing and displayed children’s writing (58% and 61%, respectively). 

Curriculum. The items that make up this dimension alternate between measuring very specifically 
use of RSL! materials (such as the LeapPad™) and capturing generally accepted good early 
childhood practices (such as using “Big Books”), organizing activities around a classroom theme, and 
using dialogic reading techniques in shared reading. On many of the items that captured RSL!- 
specific practices, we observed one-third or fewer teachers implementing the strategy (range from 
14% to 36%). The strongest areas included nearly two-thirds of teachers (61%) providing 
opportunities for language and literacy activities, including writing, that were of moderate to high 
quality, and teachers working with children one-on-one or in small groups with RSL! materials 
(67%). Lurthermore, in more than three-quarters of classrooms, teachers had made it possible 
(through arrangement of materials and prior work with children) for children to work independently 
with RSL! materials (78%). This is especially important as the electronic aids (LeapMat, LeapPad, 
LeapDesk) were intended to be used in exactly this way — as a supplementary tool to be used by 
children independently to reinforce instruction and to free the teacher to work with others individually 
or in small groups. 
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How Did The Other Classrooms In Project Upgrade Compare With RSL! Classrooms On The 
RSL! Implementation Index? 

On RSL! implementation index items that measure general good practices of early language and 
literacy instruction, BTL teachers were nearly identical to RSL! teachers (as measured by this index), 
while as one would expect, only RSL! teachers were using RSLI-specific materials (such as 
LeapPad), and thus far outscored teachers from other groups on those RSLLcurriculum-specific items 
in the Curriculum dimension of the index. 

Interaction. As noted above, all classrooms in all groups provided some full-group activities during 
the day. In the only other area rated on this dimension, similar proportions of RSL! and BTL teachers 
were observed to demonstrate positive tone and attitude towards children (58 and 53%, respectively), 
while this was true of fewer teachers in the Control group or B.E.L.L. classrooms (43 and 42%, 
respectively). 

Environment. Again, as noted above, all classrooms in all groups were arranged with learning areas 
for children that allowed children to choose materials and participate in activities independently, and 
there was more similarity than difference between groups on all items in this dimension. Interestingly, 
many more B.E.L.L. classrooms provided books and materials representing diverse cultures and 
languages (nearly 85%, in contrast with others ranging from 58-64%). Two other areas of difference 
were that more RSL! classrooms had tools available to children to practice writing words and letters 
(86%) compared with other groups (BTL the closest, with 77% of classrooms having these supports, 
and B.E.L.L. and control group classrooms closer to 70%). Close to two-thirds of RSL! and BTL 
classrooms had children’s writing on display, as compared with closer to one-third of classrooms in 
B.E.L.L. and Control group classrooms. 

Curriculum. Because so many of the items on this scale were very specifically tied to the use of RSL! 
materials (especially the electronic tools, which are unique to this curriculum), obviously, almost no 
teachers in other groups used RSL! materials. 19 However, on items such as use of Big Books, which 
any teacher could have used, teachers in more than half of BTL classrooms were observed using Big 
Books with children as opposed to only 14 and 15% of RSL! and B.E.L.L., respectively, and only one 
teacher in the Control group. 

In the more generally-accepted-best-practices items, the only difference between observed practices 
was that more teachers in both RSL! and BTL classrooms (61 and 69%, respectively, as compared 
with 42% of teachers in B.E.L.L. and Control group classrooms) provided opportunities to engage in 
literacy activities, including writing, that were of moderate to high quality. 

BTL and RSL! Implementation Indices: English-preferring vs. Spanish-preferring Teachers 

Since our impact analysis showed stronger impacts for Spanish-preferring teachers in the three 
treatment groups, it would not be surprising to find differences in implementation between Spanish- 
speaking teachers and their English-speaking counterparts. Because the sample sizes in these 
subgroups, are small, we regard all differences as suggestive only. 



19 The small number of control group classrooms that were rated as using these materials were probably 
classrooms where the teacher or director had purchased a LeapMat or LeapPad, which are both 
commercially available in toy stores, for children’s use, not as part of a curriculum. 
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BTL Implementation, By Teachers’ Language Preference 

Exhibit 4.3 shows the number and proportion of teachers, by language preference, who partially or 
fully implemented each item of the BTL implementation index. For eight out of seventeen items in 
the Reads & Discusses dimension (Comprehension within and outside reading activities and 
Vocabulary), a greater proportion of Spanish-preferring teachers implemented the activity, while on 
only one item (teachers provided vocabulary development activities outside shared reading), were a 
larger proportion of English-preferring teachers rated as implementing partially or fully. On the Oral 
Language dimension, the groups were more similar - to each other, with differences of three to five 
percentage points (which represent approximately the equivalent of one teacher) across the four 
items. 

The situation was reversed for activities that support writing. On the Writing dimension, a greater 
proportion of English-preferring teachers were rated as fully or partially implementing six out of 
fourteen items, while on only one (involving children in writing activities connected to a classroom 
theme), a larger proportion of Spanish-preferring teachers implemented the activity. On the remaining 
items in this dimension, there were only small differences (representing one to three teachers). 20 
Teachers also differed slightly in their implementation of the Classroom Culture and Routines 
dimension items, with a greater proportion of English-preferring teachers implementing two of the 
four items and a greater proportion of Spanish-preferring teachers implementing one of the four. 

RSL! Implementation, By Teachers’ Language Preference 

Exhibit 4.4 shows the number and proportion of teacher, by language preference, who implemented 
each item of the RSL! implementation index. For more than one-third of the items (4 in the 
Environment category and 4 in the Curriculum category), far more Spanish-preferring teachers 
implemented the activity. There was only one item on which more English-preferring teachers 
implemented than their Spanish-speaking counterparts (use of recommended book in shared reading); 
on the remainder of items, the trend was still for a greater proportion of Spanish-preferring teachers to 
implement as compared with their English-speaking counterparts, and there were comparable 
proportions on 5 of the items. 



20 The sample sizes in these subgroups are small — 16 teachers in the Spanish-preferring and 19 teachers in the 
English-preferring. With such small sample sizes, one teacher in sixteen represents 6 percent. 
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Exhibit 4.1 : Number and Percentage of Teachers Demonstrated Evidence of Each Item (Partially or Fully) 21 
Index, for Teachers in Each Condition 


in the BTL Implementation 


BTL Implementation Index Dimensions 


Control (n=53) 

N % 


RSL! (n=36) 
N 


% 


B.E.L.L. (n=33) 
n % 


BTL (n=35) 
n 


% 


Reads and Discusses 


Comprehension while reading aloud 


The teacher read a book aloud.* 


33 


62.3% 


32 


88.9% 


26 


78.8% 


35 


100.0% 


The teacher read aloud a BTL Book of the Week. 


I 22 


1 .9% 


0 


0.0% 


0 


0.0% 


33 


94.3% 


The teacher read aloud a book connected to a class theme. 


12 


22.6% 


16 


44.4% 


4 


12.1% 


25 


71.4% 


The teacher used one or more strategies promoting comprehension 
while reading aloud.* 


32 


60.4% 


28 


77.8% 


25 


75.8% 


31 


88.6% 


The teacher made at least 1 connection between the book being read 
and children’s experiences or class themes.* 


16 


30.2% 


21 


58.3% 


11 


33.3% 


11 


31.4% 


The teacher asked book-related questions at the end of the read aloud.* 


9 


17.0% 


7 


19.4% 


9 


27.3% 


15 


42.9% 


The average depth of post-read-aloud discussions or activities was 
moderate to “high” (e.g., they lasted at least 5 minutes and/or they 
extended or reinforced the comprehension of the book).* 


5 


9.4% 


6 


1 6.7% 


7 


21 .2% 


23 


65.7% 


Comprehension in non-reading activities 


The teacher provided one or more activities that supported the 
development of comprehension skills outside of read-alouds.* 


3 


5.7% 


7 


19.4% 


4 


12.1% 


10 


28.6% 


The average quality of discussions was moderately high or high (e.g., 
the content or topics were rich, abstract, open-ended, extended beyond 
the here-and-now, and/or related activities to children’s’ experiences).* 


3 


5.7% 


4 


11.1% 


6 


18.2% 


6 


17.1% 



21 Item Scores: For most items, a teacher scored one point if she or he engaged in the described instructional activity. Items with an asterisk (*) were scored as 
follows: “0” if the teacher never engaged in the instructional activity, “1” if she engaged in the activity with a small or a large group, and “2” if she engaged in 
the activity with both a small and a large group. 

~ 2 This variable was created by checking the title of the book read with a list of BTL Book-of-the-Week (BOW), so in this single classroom, a non-BTL teacher was 
observed reading a McGraw-Hill book that is on the BTL BOW list. 
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Exhibit 4.1 : Number and Percentage of Teachers Demonstrated Evidence of Each Item (Partially or Fully) 21 
Index, for Teachers in Each Condition 


in the BTL Implementation 


BTL Implementation Index Dimensions 


Control (n=53) 

N % 


RSL! (n=36) 

N % 


B.E.L.L. (n=33) 
n % 


BTL (n=35) 
n 


% 


Vocabulary 


The teacher emphasized vocabulary in at least one read aloud.* 


10 


18.9% 


9 


25.0% 


7 


21.2% 


7 


20.0% 


The quality of story-related vocabulary instruction during read alouds 
was moderate to high (e.g., one or more vocabulary word was discussed 
and a comprehension support was used). 


5 


9.4% 


8 


22.2% 


4 


12.1% 


5 


14.3% 


The teacher provided one or more activities that supported vocabulary 
development outside of read-alouds (e.g., vocabulary knowledge was 
provided in the context of explanations, writing, songs, stories, rhymes, 
language games, discussions, shared reading, emergent writing, and/or 
child tagging or matching).* 


5 


9.4% 


13 


36.1% 


2 


6.1% 


5 


14.3% 


Oral language 


The teacher provided at least one literacy activity aimed at developing 
children’s oral communication and/or listening skills. 


21 


39.6% 


23 


63.9% 


22 


66.7% 


21 


60.0% 


The teacher had students engage in language games, rhymes, songs, 
storytelling, or discussion in order to comprehend a text. If the activity 
was a discussion, it was rated moderately high or high (e.g., they 
included turn-taking by students and the teacher elaborated or asked 
students to elaborate on comments).* 


0 


0.0% 


0 


0.0% 


0 


0.0% 


0 


0.0% 


The teacher used one or more instructional strategies that promoted 
higher order thinking during or after a read-aloud session (e.g., they 
asked book-related open-ended questions that required speculation, 
and/or expanded responses, thinking, or analyses).* 


21 


39.6% 


17 


47.2% 


16 


48.5% 


19 


54.3% 


The teacher provided at least a few opportunities for oral language. 


36 


67.9% 


29 


80.6% 


26 


78.8% 


27 


77.1% 


The average quality of the oral language activities provided by the 
teacher was moderate to high (e.g., the teacher provided integrated, 
higher quality oral language opportunities of varying types for most 
students individually or in small groups in which the teacher scaffolded or 
extended discussions with multiple turns that focused on non- 
management topics). 


28 


52.8% 


22 


61.1% 


19 


57.6% 


25 


71.4% 
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Exhibit 4.1 : Number and Percentage of Teachers Demonstrated Evidence of Each Item (Partially or Fully) 21 in the BTL Implementation 
Index, for Teachers in Each Condition 



BTL Implementation Index Dimensions 


Control (n=53) 

N % 


RSL! (n=36) 

N % 


B.E.L.L. (n=33) 
n % 


BTL (n=35) 
n 


% 


Writing 


Examples of children’s writing were on display in the classroom (other 
than just names). 


20 


37.7% 


22 


61.1% 


12 


36.4% 


24 


68.6% 


Children spent time in journal-writing activities. 


0 


0.0% 


5 


13.9% 


1 


3.0% 


0 


0.0% 


The teacher used at least one high quality activity aimed at developing 
knowledge of the functions and features of print (e.g., using authentic 
print materials, graphic organizers, or word webs). 


0 


0.0% 


0 


0.0% 


0 


0.0% 


2 


5.7% 


The teacher provided a separate writing area and/or writing materials 
accessible to children. 


22 


41 .5% 


18 


50.0% 


13 


39.4% 


16 


45.7% 


In at least some writing activities, the writing was done by the children 
themselves rather than by adults. 


31 


58.5% 


27 


75.0% 


20 


60.6% 


27 


77.1% 


The teacher sometimes, often, or consistently encouraged children to 
write on their own rather than insisting on conventional letter formation or 
spelling. 


28 


52.8% 


22 


61.1% 


19 


57.6% 


27 


77.1% 


The teacher provided one or more literacy activities that involved 
emergent writing, copying, or tracing. 


29 


54.7% 


27 


75.0% 


15 


45.5% 


26 


74.3% 


Children were engaged in one or more writing activity (including 
emergent writing, copying, and tracing) that was connected to a class 
theme during the day. 


3 


5.7% 


2 


5.6% 


1 


3.0% 


5 


14.3% 


The teacher worked with most or all of the students in writing activities 
during the day. 


25 


47.2% 


19 


52.8% 


13 


39.4% 


27 


77.1% 


Writing activities and opportunities were sometimes, often, or 
consistently conducted with individuals or small groups. 


27 


50.9% 


26 


72.2% 


15 


45.5% 


23 


65.7% 


The teacher provided two or more opportunities to engage in writing. 


24 


45.3% 


22 


61.1% 


14 


42.4% 


25 


71.4% 


The teacher provided three or more types of writing activities. 


9 


1 7.0% 


12 


33.3% 


5 


1 5.2% 


12 


34.3% 


The teacher provided children with writing opportunities that were 


31 


58.5% 


23 


63.9% 


19 


57.6% 


29 


82.9% 



sometimes, often, or consistently of higher quality (e.g., emergent 
writing, captioning, dictating, writing names on work, book-making, 
and/or writing in journals. 
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Exhibit 4.1 : Number and Percentage of Teachers Demonstrated Evidence of Each Item (Partially or Fully) 21 in the BTL Implementation 
Index, for Teachers in Each Condition 



BTL Implementation Index Dimensions 


Control (n=53) 

N % 


RSL! (n=36) 

N % 


B.E.L.L. (n=33) 
n % 


BTL (n=35) 

n % 


The teacher provided writing opportunities that were sometimes or often 
integrated into activities with goals other than literacy. 


23 


43.4% 


17 


47.2% 


10 


30.3% 


15 


42.9% 


Individualized Software Instruction 


The average child used a computer for the amount of time 
recommended by BTL, or longer (pre-K: 8-12, K: 12-15 minutes). 


7 


13.2% 


4 


11.1% 


2 


6.1% 


20 


57.1% 


Classroom Culture and Routines 


The classroom was well-organized and had the space and layout to 
afford children a variety of independent activity choices, including 
learning centers (e.g. it had at least one distinct activity center, some 
materials that were marked, sorted, and stored; the layout allowed for at 
least two group sizes, some independent choices for students, sufficient 
space, adequate light, and/or no odors). 


53 


1 00.0% 


36 


1 00.0% 


33 


1 00.0% 


35 


100.0% 


The average child spent less than 33% of the time in transitions, 
routines, and management or being uninvolved in activities. 


42 


79.2% 


34 


94.4% 


27 


81.8% 


29 


82.9% 


The average child spent at least 50% of their time engaged in 
educationally “high value” activities (including: reading, alphabet, oral 
language, sounds, writing, science/nature, math concepts, dramatic play, 
creative play, block play, and fine motor play). 


21 


39.6% 


13 


36.1% 


9 


27.3% 


11 


31.4% 


The average child spent at least 20% of their time in groups of 5 or fewer 


39 


73.6% 


22 


61.1% 


22 


66.7% 


20 


57.1% 



(not including meals, routines, transitions, or management). 

a Item Scores: For most items, a teacher scored one point if she or he engaged in the described instructional activity. Items with a were scored as follows: “0” of the teacher 
never engaged in the instructional activity, “1” if she or he engaged in the activity with a small or a large group, and “2” if she or he engaged in the activity with both a small and a 
large group. 

EXHIBIT READS: In five BTL classrooms (14.3 percent of the Project Upgrade BTL sample), children were engaged in one or more writing activity (including emergent writing, 
copying, and tracing) that was connected to a class theme during the day; in contrast, in three Control group classrooms (5.7 percent of the Project Upgrade Control group sample), 
children were engaged in such activities. 

Source: Project Upgrade, spring 2005. 
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Table 4.2: Number and Percentage of Teachers who Demonstrated Evidence of Each Item in the RSL! Implementation Index, for 
Teachers in each Condition 




Control (n=53) 


RSL! (n=36) 


B.E.L.L. (n=33) 


BTL (n=35) 


RSL! Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


N 


% 


Interaction 


The tone and classroom conversations are positive; 
teacher shows respect for children and listens 
attentively, encouraging them to learn likewise. 


23 


43.4% 


21 


58.3% 


14 


42.4% 


20 


57.1% 


Evidence of full-group activities, where each child is 
encouraged to participate. 


53 


1 00.0% 


36 


100.0% 


33 


100.0% 


35 


100.0% 


Environment 


In the classroom: Furnishings and materials are 
appropriately sized, in good repair, clearly organized, 
appealing, accessible and not crowded. 


46 


86.8% 


26 


72.2% 


26 


78.8% 


25 


71 .4% 


Learning areas provide opportunities for children to 
engage independently in activities. 


53 


1 00.0% 


36 


100.0% 


33 


100.0% 


35 


1 00.0% 


An inviting and orderly area is set aside for book 
reading, made comfortable by soft materials such as 
pillows and cushions. 


37 


69.8% 


22 


61.1% 


20 


60.6% 


12 


34.3% 


Books promote cultural diversity, and there are 
materials in other languages that represent 
children’s culture of origin. 


34 


64.2% 


21 


58.3% 


28 


84.8% 


22 


62.9% 


There are word cards, with names or familiar words 
intended to support children’s writing. 


53 


1 00.0% 


35 


97.2% 


33 


100.0% 


35 


1 00.0% 


There are toys and materials accessible to children 
that include words/letters and there are tools 
accessible to children to help them practice writing 
words/letters (e.g., stencils, templates, tracing 
sheets). 


38 


71 .7% 


31 


86.1% 


23 


69.7% 


27 


77.1% 


There is a variety of paper and writing tools available 
(chalkboard, pencils, crayons, etc.). 


26 


49.1% 


19 


52.8% 


17 


51 .5% 


20 


57.1% 
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Table 4.2: Number and Percentage of Teachers who Demonstrated Evidence of Each Item in the RSL! Implementation Index, for 
Teachers in each Condition 




Control (n=53) 


RSL! (n=36) 


B.E.L.L. (n=33) 


BTL (n=35) 


RSL! Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


N 


% 


Variety of teacher's captioning (from child's work) 
and children’s own writing (journals, sign in, etc.) on 
display. 


20 


37.7% 


22 


61.1% 


12 


36.4% 


24 


68.6% 


RSL! Curriculum 


There is evidence in the classroom of the use of 
RSL! materials. 


3 


5.7% 


33 


91.7% 


1 


3.0% 


0 


0.0% 


Opportunities for extended uses of language, 
(activities, materials, and displays) are coordinated 
with RSL! goals in each unit and its current theme. 


14 


26.4% 


13 


36.1% 


5 


15.2% 


16 


45.7% 


Teacher encourages the use of the LeapMat to 
review the letters learned in the current lesson, and 
follows suggested activities in the “Teacher’s 
Manual”. 


0 


0.0% 


11 


30.6% 


0 


0.0% 


0 


0.0% 


Teacher integrates the LeapPad, the correspondent 
book, and activities (“Teacher's Manual”) with the 
current lesson. 


2 


3.8% 


10 


27.8% 


0 


0.0% 


0 


0.0% 


Teacher encourages the use of the LeapDesk, and 
models its use to review letters and numbers as 
suggested in the Manual. 


0 


0.0% 


8 


22.2% 


0 


0.0% 


0 


0.0% 


Teacher reads the suggested book for the lesson 
(Read-Aloud Library), and implements suggested 
activities in the “Teacher’s Resource Guide”. 


9 


1 7.0% 


11 


30.6% 


0 


0.0% 


2 


5.7% 


The teacher uses the Big Books and Read Aloud 
Library for full-group and small group activities 
according to the lesson. 


1 


1 .9% 


5 


13.9% 


5 


15.2% 


19 


54.3% 


Teaching strategies support reading comprehension, 
questioning, predicting, summarizing, problem- 
solving, and reflecting on learning. 


24 


45.3% 


17 


47.2% 


14 


42.4% 


17 


48.6% 
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Table 4.2: Number and Percentage of Teachers who Demonstrated Evidence of Each Item in the RSL! Implementation Index, for 
Teachers in each Condition 





Control (n=53) 


RSL! (n=36) 


B.E.L.L. (n=33) 


BTL (n=35) 


RSL! Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


N 


% 


Adults engage in one-to-one or small group activities 
using RSL! materials. 


1 


1 .9% 


24 


66.7% 


0 


0.0% 


0 


0.0% 


Adults facilitate opportunities and model for the 
children to practice their reading and writing skills as 
learned in the lessons. 


22 


41 .5% 


22 


61.1% 


14 


42.4% 


24 


68.6% 


Children use RSL! materials on their own or with a 
friend by own initiative. 


3 


5.7% 


28 


77.8% 


1 


3.0% 


0 


0.0% 
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Table 4.3: Number and Percentage of BTL Teachers Implementing Each Item in the BTL Implementation Index Dimensions, By 
Teacher's Language Preference 

BTL BTL 



Spanish-Speaking English-Speaking 

Teachers Teachers BTL Full Group 

(n=16) (n=19) (n=35) 



BTL Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


Reads and Discusses 


Comprehension while reading aloud 


The teacher read a book aloud.* 


16 


100.0% 


19 


1 00.0% 


35 


100.0% 


The teacher read aloud a BTL Book of the Week. 


16 


100.0% 


17 


89.5% 


33 


94.3% 


The teacher read aloud a book connected to a class theme. 


14 


87.5% 


11 


57.9% 


25 


71.4% 


The teacher used one or more strategies promoting comprehension while 
reading aloud.* 


15 


93.8% 


16 


84.2% 


31 


88.6% 


The teacher made at least 1 connection between the book being read and 
children’s experiences or class themes.* 


5 


31.3% 


6 


31.6% 


11 


31.4% 


The teacher asked book-related questions at the end of the read aloud.* 


7 


43.8% 


8 


42.1% 


15 


42.9% 


The average depth of post-read-aloud discussions or activities was moderate to 
“high” (e.g., they lasted at least 5 minutes and/or they extended or reinforced the 
comprehension of the book).* 


12 


75.0% 


11 


57.9% 


23 


65.7% 


Comprehension in non-reading activities 


The teacher provided one or more activities that supported the development of 
comprehension skills outside of read-alouds.* 


6 


37.5% 


4 


21.1% 


10 


28.6% 


The average quality of discussions was moderately high or high (e.g., the content 
or topics were rich, abstract, open-ended, extended beyond the here-and-now, 
and/or related activities to children’s' experiences).* 


4 


25.0% 


2 


1 0.5% 


6 


17.1% 


Vocabulary 


The teacher emphasized vocabulary in at least one read aloud.* 


4 


25.0% 


3 


1 5.8% 


7 


20.0% 


The quality of story-related vocabulary instruction during read alouds was 
moderate to high (e.g., one or more vocabulary word was discussed and a 
comprehension support was used). 


3 


18.8% 


2 


1 0.5% 


5 


14.3% 
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Table 4.3: Number and Percentage of BTL Teachers Implementing Each Item in the BTL Implementation Index Dimensions, By 
Teacher's Language Preference 






BTL 




BTL 








Spanish-Speaking 

Teachers 

(n=16) 


English-Speaking 

Teachers 

(n=19) 


BTL Full Group 
(n=35) 


BTL Implementation Index Dimensions 


N % 


N 


% 


N 


% 


The teacher provided one or more activities that supported vocabulary 
development outside of read-alouds (e.g., vocabulary knowledge was provided in 
the context of explanations, writing, songs, stories, rhymes, language games, 
discussions, shared reading, emergent writing, and/or child tagging or 
matching).* 


1 6.3% 


4 


21.1% 


5 


14.3% 


Oral language 


The teacher provided at least one literacy activity that afforded students 
knowledge of oral communication and/or listening skills. 


1 0 62.5% 


11 


57.9% 


21 


60.0% 


The teacher provided one or more activities that included language games, 
rhymes, songs, storytelling, or discussion; that afforded students knowledge in 
the comprehension of a text; that included an average depth of the discussion 
that was moderately high or high (e.g., they included turn-taking by students and 
the teacher elaborated or asked students to elaborate on comments).* 


0 0.0% 


0 


0.0% 


0 


0.0% 


The teacher used one or more instructional strategies that promoted higher order 
thinking during or after a read-aloud session (e.g., they asked book-related open- 
ended questions that required speculation, and/or expanded responses, thinking, 
or analyses).* 


9 56.3% 


10 


52.6% 


19 


54.3% 


The teacher provided at least a few opportunities for oral language. 


12 75.0% 


15 


78.9% 


27 


77.1% 


The average quality of the oral language activities provided by the teacher was 
moderate to high (e.g., the teacher provided integrated, higher quality oral 
language opportunities of varying types for most students individually or in small 
groups in which the teacher scaffolded or extended discussions with multiple 
turns that focused on non-management topics). 


1 1 68.8% 


14 


73.7% 


25 


71.4% 


Writing 


Examples of children’s writing were on display in the classroom (other than just 
names). 


1 1 68.8% 


13 


68.4% 


24 


68.6% 


Children spent time in journal-writing activities. 


0 0.0% 


0 


0.0% 


0 


0.0% 
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Table 4.3: Number and Percentage of BTL Teachers Implementing Each Item in the BTL Implementation Index Dimensions, By 
Teacher's Language Preference 

BTL BTL 



Spanish-Speaking English-Speaking 

Teachers Teachers BTL Full Group 

(n=16) (n=19) (n=35) 



BTL Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


The teacher used at least one high quality activity aimed at developing 
knowledge of the functions and features of print (e.g., using authentic print 
materials, graphic organizers, or word webs). 


1 


6.3% 


1 


5.3% 


2 


5.7% 


The teacher provided a separate writing area and/or writing materials accessible 
to children. 


5 


31.3% 


11 


57.9% 


16 


45.7% 


In at least some writing activities, the writing was done by the children 
themselves rather than by adults. 


11 


68.8% 


16 


84.2% 


27 


77.1% 


The teacher sometimes, often, or consistently encouraged children to write on 
their own rather than insisting on conventional letter formation or spelling. 


10 


62.5% 


17 


89.5% 


27 


77.1% 


The teacher provided one or more literacy activities that involved emergent 
writing, copying, or tracing. 


12 


75.0% 


14 


73.7% 


26 


74.3% 


Children were engaged in one or more writing activity (including emergent 
writing, copying, and tracing) that was connected to a class theme during the 
day. 


3 


18.8% 


2 


1 0.5% 


5 


14.3% 


The teacher worked with most or all of the students in writing activities during the 
day. 


11 


68.8% 


16 


84.2% 


27 


77.1% 


Writing activities and opportunities were sometimes, often, or consistently 
conducted with individuals or small groups. 


10 


62.5% 


13 


68.4% 


23 


65.7% 


The teacher provided two or more opportunities to engage in writing. 


11 


68.8% 


14 


73.7% 


25 


71 .4% 


The teacher provided three or more types of writing activities. 


6 


37.5% 


6 


31.6% 


12 


34.3% 


The teacher provided children with writing opportunities that were sometimes, 
often, or consistently of higher quality (e.g., emergent writing, captioning, 
dictating, writing names on work, book-making, and/or writing in journals. 


11 


68.8% 


18 


94.7% 


29 


82.9% 


The teacher provided writing opportunities that were sometimes or often 
integrated into activities with goals other than literacy. 


6 


37.5% 


9 


47.4% 


15 


42.9% 
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Table 4.3: Number and Percentage of BTL Teachers Implementing Each Item in the BTL Implementation Index Dimensions, By 
Teacher's Language Preference 

BTL BTL 



Spanish-Speaking English-Speaking 

Teachers Teachers BTL Full Group 

(n=16) (n=19) (n=35) 



BTL Implementation Index Dimensions 


N 


% 


N 


% 


N 


% 


Individualized Software Instruction 


The average child used a computer for the amount of time recommended by 
BTL, or longer (pre-K: 8-12, K: 12-15 minutes). 


10 


62.5% 


10 


52.6% 


20 


57.1% 


Classroom Culture and Routines 


The classroom was well-organized and had the space and layout to afford 
children a variety of independent activity choices, including learning centers (e.g. 
it had at least one distinct activity center, some materials that were marked, 
sorted, and stored; the layout allowed for at least two group sizes, some 
independent choices for students, sufficient space, adequate light, and/or no 
odors). 


16 


100.0% 


19 


1 00.0% 


35 


100.0% 


The average child spent less than 33% of the time in transitions, routines, and 
management or being uninvolved in activities. 


12 


75.0% 


17 


89.5% 


29 


82.9% 


The average child spent at least 50% of their time engaged in educationally “high 
value” activities (including: reading, alphabet, oral language, sounds, writing, 
science/nature, math concepts, dramatic play, creative play, block play, and fine 
motor play). 


4 


25.0% 


7 


36.8% 


11 


31.4% 


The average child spent at least 20% of their time in groups of 5 or fewer (not 


10 


62.5% 


10 


52.6% 


20 


57.1% 



including meals, routines, transitions, or management). 

EXHIBIT READS: In five Project Upgrade classrooms (14.3 percent of the Project Upgrade sample), children were engaged in one or more writing activity (including 
emergent writing, copying, and tracing) that was connected to a class theme during the day 

Sources: Project Upgrade, spring 2005. 
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Table 4.4: Number and Percentage of Teachers Who Demonstrated Evidence of Each Item in the RSL! Instructional Dimensions in 
Project Upgrade, for English-Speaking Teachers, Spanish-Speaking Teachers, and the Full RSL! Group 

RSL! RSL! 



Spanish-Speaking English-Speaking 

Teachers Teachers RSL! Full Group 

(n=17) (n=19) (n=36) 





N 


% 


N 


% 


N 


% 


Interaction 


The tone and classroom conversations are positive; teacher shows respect for 
children and listens attentively, encouraging them to learn likewise. 


10 


58.8% 


11 


57.9% 


21 


58.3% 


Evidence of full-group activities, where each child is encouraged to participate. 


17 


100.0% 


19 


1 00.0% 


36 


1 00.0% 


Environment 


In the classroom: Furnishings and materials are appropriately sized, in good 
repair, clearly organized, appealing, accessible and not crowded. 


15 


88.2% 


11 


57.9% 


26 


72.2% 


Learning areas provide opportunities for children to engage independently in 
activities. 


17 


100.0% 


19 


1 00.0% 


36 


100.% 


An inviting and orderly area is set aside for book reading, made comfortable by 
soft materials such as pillows and cushions. 


11 


64.7% 


11 


57.9% 


22 


61.1% 


Books promote cultural diversity, and there are materials in other languages that 
represent children's culture of origin. 


15 


88.2% 


6 


31.6% 


21 


58.3% 


There are word cards, with names or familiar words intended to support 
children’s writing. 


17 


100.0% 


18 


94.7% 


35 


97.2% 


There are toys and materials accessible to children that include words/letters, 
and there are tools accessible to children to help them practice writing 
words/letters (e.g., stencils, templates, tracing sheets). 


15 


88.2% 


16 


84.2% 


31 


86.1% 


There is a variety of paper and writing tools available (chalkboard, pencils, 
crayons, etc.). 


10 


58.8% 


9 


47.4% 


19 


52.8% 


Variety of teacher’s dictation (teacher-led activity, from child's work) and 
children's writing (journals, sign in, etc) on display. 


12 


70.6% 


10 


52.6% 


22 


61.1% 


RSL! Curriculum 


There is evidence in the classroom of the use of RSL! materials. 


16 


94.1% 


17 


89.5% 


33 


91.7% 



•c* 
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Table 4.4: Number and Percentage of Teachers Who Demonstrated Evidence of Each Item in the RSL! Instructional Dimensions in 
Project Upgrade, for English-Speaking Teachers, Spanish-Speaking Teachers, and the Full RSL! Group 

RSL! RSL! 



Spanish-Speaking English-Speaking 

Teachers Teachers RSL! Full Group 

(n=17) (n=19) (n=36) 





N 


% 


N 


% 


N 


% 


Opportunities for extended uses of language, (activities, materials, and displays) 
are coordinated with RSL! goals in each unit and its current theme. 


7 


41.2% 


6 


31.6% 


13 


36.1% 


Teacher encourages the use of the LeapMat to review the letters learned in the 
current lesson, and follows suggested activities in the “Teacher’s Manual”. 


5 


29.4% 


6 


31.6% 


11 


30.6% 


Teacher integrates the LeapPad, the correspondent book, and activities 
(“Teacher’s Manual”) with the current lesson. 


5 


29.4% 


5 


26.3% 


10 


27.8% 


Teachers encourage the use of the LeapDesk, and models its use to review 
letters and numbers as suggested in the Manual. 


2 


1 1 .8% 


6 


31.6% 


8 


22.2% 


Teacher reads the suggested book for the lesson (Read-Aloud Library), and 
implements suggested activities in the “Teacher’s Resource Guide”. 


4 


23.5% 


7 


36.8% 


11 


30.6% 


The teacher uses the Big Books and Read Aloud Library for full-group and small 
group activities according to the lesson. 


2 


1 1 .8% 


3 


1 5.8% 


5 


13.9% 


Teaching strategies support reading comprehension, questioning, predicting, 
summarizing, problem-solving, and reflecting on learning. 


9 


52.9% 


8 


42.1% 


17 


47.2% 


Adults engage in one-to-one or small group activities using RSL! materials. 


12 


70.6% 


12 


63.2% 


24 


66.7% 


Adults facilitate opportunities and model for the children to practice their reading 
and writing skills as learned in the lessons. 


11 


64.7% 


11 


57.9% 


22 


61.1% 


Children use RSL! materials on their own or with a friend by own initiative. 


14 


82.4% 


14 


73.7% 


28 


77.8% 


EXHIBIT READS: In 21 RSL! classrooms (58.3% percent of the RSL! sample), caregivers speak warmly and listen attentively to children. 








Sources: Project Upgrade, spring 2005. 
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Chapter Five: Discussion 



In this report we examined two distinct aspects of the implementation of interventions tested in 
Project Upgrade: a single professional development model; and three curriculum models. 

Implementation of the Professional Development Model 

It seems fair to conclude that the professional development model was well-implemented : 23 training 
sessions and ongoing mentoring were delivered at least as often as the design called for, by well- 
trained staff, and there was little turnover of staff. The coaches and mentors for each intervention 
were highly-motivated and persistent in their efforts with classroom staff. 

However, absolute fidelity to a model is neither easy to achieve nor necessarily desirable. The 
curriculum developers, coaches and mentors had to adapt the plan to meet a set of challenges. These 
adaptations did not result in deviation from the principles underlying the model. Rather, in all cases, 
meeting the challenges resulted in an intensification of effort on the paid of all three groups. 

To meet the challenge of teacher turnover, developers and coaches provided additional training 
sessions and more frequent visits to the center by coaches and mentors. When teachers were unable to 
attend follow-up training sessions because there was no substitute teacher available, additional, 
smaller training sessions were organized. To meet the challenge of teachers who lacked basic 
classroom management skills, mentors made more frequent visits, reorganized classroom space and 
advised teachers on classroom management, while continuing to address the requirement to 
implement the curriculum. 

To meet the challenge of director turnover, or changes in center ownership, coaches and the Project 
Coordinator visited the centers repeatedly to explain the study, the intervention that the center was 
participating in, and the benefits and obligations of participation. As a result, attrition of centers from 
the study was very low. 

While most of these challenges are to be expected in any subsequent implementation of these models, 
some are a by-product of the evaluation in which the interventions were nested, which dictated the 
timeframe for implementation of the interventions. The challenge of teachers who lacked classroom 
management skills was addressed by increasing the frequency and intensity of visits. In a replication 
of the model, a better solution (advocated by developers, coaches and mentors) would be to delay 
training in curriculum implementation until the issue of basic skills had been addressed. Similarly, 
teachers who were essentially monolingual in Spanish struggled to honor the request to conduct 
literacy activities in English and needed manuals and other materials in Spanish, beyond what the 
developers had anticipated. However, in training sessions, the teachers themselves suggested that they 
would have welcomed classes in English, as preparation for the interventions. 

Another challenge for coaches and mentors was the number of teachers who resisted implementing a 
curriculum, either because they had not wanted to participate in the study at all (the directors had 



23 The curriculum trainers observed each mentor regularly and gave them feedback. However, there was no 
formal rating of mentor fidelity. 



Project Upgrade Implementation Study 



46 





imposed the decision on them) or preferred a different curriculum. Coaches and mentors spent a great 
deal of additional time working with these teachers, sometimes to little effect. Again, this additional 
effort was helpful for the study, in reducing attrition and strengthening implementation, but the 
problem would either not arise or would be approached differently in a replication. In a replication, 
program managers might encourage active participation of teachers in the design of a professional 
development and curriculum strategy, or might work to increase interest and motivation before 
beginning training. 

A final note about the professional development model and the effect of the evaluation on it concerns 
the duration of mentoring and support. These activities, in almost all cases, stopped at the end of the 
18-month period of the interventions. Many teachers requested some continuing support and a 
number clearly needed it. In a larger-scale implementation, it is likely that mentoring support would 
be more tailored to the needs of the individual (always maintaining the basic model), and sustained on 
a lower level, tapering off rather than being abruptly withdrawn. 

Implementation of the Curriculum Models 

To guide their own supportive efforts, mentors and coaches used quite detailed fidelity measures that 
were curriculum-specific (although, as our subsequent efforts show, they also shared elements of 
good, research-supported teaching practice). To provide the study with a simpler common assessment 
of level of implementation, the mentors rated classrooms on a five-point scale. Ultimately, this told us 
little beyond the fact that most teachers in the three treatment groups were implementing the curricula 
satisfactorily (only a few received a score of five on the five-point scale, while many received a score 
of four, meaning “fairly well-implemented”). Although the developers could analyze their own 
fidelity measures to identify areas of weakness that might need attention, the global ratings did not 
provide that kind of information. 

The implementation indexes developed post hoc were imperfect because it was impossible to capture 
all of the behaviors and practices that were specifically linked to individual curricula. Nevertheless, 
the measures were useful. First, the comparisons across all four groups made us aware that, although 
teachers were not always effectively implementing finer-grained and curriculum-specific practices, a 
majority of teachers in both the BTL and RSL! groups were implementing practices and activities that 
research has linked to the development of language and literacy, and that formed the basis of each 
curriculum. By contrast, on many items, B.E.L.L. teachers more closely resembled teachers in the 
control group. Although in both of these groups (B.E.L.L. and control) some teachers were engaging 
children in activities that support language and literacy development, the proportions were 
considerably smaller than in the other two groups. 24 

As an example of how this worked in practice, consider that all BTL teachers read a book aloud 
during the observation, and between 62 percent (control group) and 89 percent (RSL) did so. Slightly 
lower percentages used one or more strategies to promote comprehension during the read-aloud. Both 
of these practices arc supported as “good practice” and training in any literacy curriculum would 



~ 4 B.E.L.L. was delivered as two brief (15-20) minute activities, both of which were always scheduled in the 
morning, which was when our four-hour classroom observations were conducted. Observations across all 
study groups probably missed some more informal literacy activities and opportunities that may have 
occurred later in the day, after children had napped and before they were picked up. 
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begin with these. By contrast, scores on a BTL curriculum-specific item that extends the book 
discussion (teacher asked book-related questions at the end of the read-aloud), scores were lower, 
even for BTL teachers (43%), and lower still in the other groups (17% for the control group teachers, 
19% for RSL and 27% for B.E.L.L.). Similarly, in supporting writing activities, a focus of both BTL 
and RSL, high proportions of teachers consistently encouraged children to write on their own (77% in 
BTL, 61% in RSL, 57% for B.E.L.L. and 53% in the control group). However, for an item that 
emphasized repeated opportunities for children to engage in writing, about one-third of BTL and RSL 
teachers were scored as doing this, with much lower proportions in the other two groups (15% in 
B.E.L.L. and 17% in the control group). What we understand from these findings is that teachers 
were variably successful in implementing practices that increased the quality and intensity of a 
general good practice, and that were part of the curriculum for which they were trained. But, in the 
absence of guidance (in B.E.L.L. and the control group) few if any teachers implemented them. 

We note again that we were unable to create an implementation index for B.E.L.L. that reflected the 
idiosyncratic set of practices mandated by the curriculum. However, the relatively lower usage by a 
large proportion of B.E.L.L. teachers of the more general practices that are strongly linked in the 
research literature to later reading success provides a possible explanation for the intervention’s 
failure to produce impacts on children. 25 

Areas in which many teachers struggled included: activities to support children’s independent 
writing; extending support for comprehension and vocabulary support throughout the day and outside 
specific literacy activities; and working with small groups of children or individuals. Because the 
classes were small on average, this latter may have seemed less important to teachers, but it also 
reflects a common difficulty that teachers in all preschool settings experience. The ability to work 
one-on-one or in small groups with children is critically dependent on the teacher’ s skill in structuring 
the small-group activities so that other children are productively engaged. 



25 It should also be remembered that the B.E.L.L. intervention consisted of two brief ( 15-minute) activities, 
one almost totally focused on phonological awareness. If phonological awareness is less strongly linked to 
literacy outcomes than, say, vocabulary or writing, then the B.E.L.L. strategy could be ineffective, because 
it failed to include time for these other activities. 
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Appendix A 

Creation of Implementation Indexes 
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Creation of Implementation Indexes 

The development of the implementation index for each curriculum involved three major steps: 

1. We used the developer’s fidelity checklist as a stalling point for identifying the instructional 
practices that constitute fidelity to the curriculum. 

2. We linked each instructional practice required for fidelity to the curriculum with data 
collected using the OMLIT, if possible (i.e., a crosswalk between the curriculum concepts 
and OMLIT variables to match OMLIT variables with specific instructional practices 
required for fidelity); 

3. For each instructional practice that could be mapped to data from the OMLIT, we set criteria 
for implementation fidelity. 



We describe each step in greater detail below. The process of developing the fidelity measure using 
data from the OMLIT was successful for BTL and RSL! Flowever, we could not create a measure of 
B.E.L.L. implementation fidelity using the OMLIT data, because the OMLIT data did not map onto 
the unique B.E.L.L. instructional practices. 

Step 1: Identifying Core Instructional Practices using the Fidelity Checklist. We started the 
process of identifying key practices for each curriculum by examining the curriculum developer’s 
fidelity checklist. To identify BTL practices, we examined the Breakthrough to Literacy Fidelity 
Instrument: 2006-2007, which was developed by Abt in collaboration with the BTL developers and 
instructional coaches for another study of BTL. 26 For RSL!, we examined the RSL! Language and 
Literacy Observation Toot, and for B.E.L.L., we examined the B.E.L.L. Observation Checklist. For 
each curriculum, the developer’ s fidelity instrument identifies the key instructional practices that 
teachers are expected to use. The fidelity instruments were intended to be used by coaches to 
document implementation in study classrooms. The global ratings by coaches using the developers’ 
fidelity instruments did not yield enough variation for us to use in our analysis. In a final effort to 
examine implementation, we measured as many of the core elements as possible based on our own 
research team’s observations conducted using the OMLIT. 

Step 2: Linking the Core Instructional Practices to OMLIT variables. For each curriculum, we 
identified the key concepts with which the OMLIT variables align most closely. Though not all of the 
categories included on each fidelity instrument could be linked with OMLIT variables, most elements 
deemed essential to BTL and to RSL! could be translated into variables from data collected by the 
research team using the OMLIT. For each curriculum, the instructional practices measured by the 
developer’ s fidelity checklist and the corresponding construct that was measured by the OMLIT are 
presented in Exhibits Al, A2, and A3. A crosswalk is presented for BTL in Exhibit Al, for RSL! in 
Exhibit A2, and for B.E.L.L. in Exhibit A3. Core instructional practices for the curriculum are 
described in the left-hand column and the corresponding behavior, characteristic, or rating measured 
by the OMLIT is identified in the right-hand column of the same row. Instructional practices and 
characteristics from the fidelity checklists that did not align with OMLIT variables arc marked “Not 
measured”. 



26 Although that fidelity checklist was organized differently from the checklist used by mentors in Project 
Upgrade, the same elements were present in both. 
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The OMLIT items used to measure implementation of the curricula were taken from the following 
OMLIT instruments: 

• CLIP: Classroom Literacy Instruction Profile 

• SNAP: Snapshot of Classroom Activities 

• RAP: Read-Aloud Profile 

• CLOC: Classroom Literacy Opportunities Checklist 

• QUILL: Quality of Language and Literacy Instruction 

There are five key categories of BTL practices: Reads & Discusses; Daily Writing; Individualized 
Software Instruction; Classroom Culture and Management Routines; and Reads, Writes, and Talks at 
Home. Four of the five can be measured using OMLIT data. The specific instructional practices that 
comprise each dimension arc shown in Exhibit Al. 

There are five key categories of the RSL! curriculum: Interaction, Environment, RSL! Curriculum, 
Assessment, and Family Involvement. Three of the five RSL! categories could be measured with data 
from the OMLIT. The specific instructional practices that comprise each dimension arc shown in 
Exhibit A2. 

The B.E.L.L. curriculum is organized around two main categories — Reading/Oral Language and Print 
Awareness/Phonological Awareness. Each of those is subdivided into subcategories. The specific 
instructional practices for each category are shown and, where possible, corresponding OMLIT 
variables are listed in Exhibit A3. Because such a small proportion of items could be translated 
accurately into OMLIT variables, we were not able to create an index that would provide a fair 
representation of the B.E.L.L. curriculum. 

Step 3: Setting Criteria for Evidence of Implementation Fidelity. For each instructional practice 
that could be measured using variables from the OMLIT, we set criteria for evidence of fidelity. For 
example, one BTL instructional practice involves engaging children in post-reading discussions of 
moderate or greater depth. In order for a teacher to have demonstrated evidence of this practice, she 
needed to have a rating of 3 or higher (out of 5) on a Read Aloud Profile (RAP) quality rating of the 
depth of post-reading activity. In another example involving an RSL! instructional practice, teachers 
are supposed to facilitate opportunities for students to practice reading and writing. In order to have 
demonstrated evidence of this practice, teachers had to have ratings above 2 (out of 5) for both the 
frequency and quality of reading and writing activities measured by the QUILL. 

For each instructional practice that could be measured using variables from the OMLIT, we set 
criteria for evidence of fidelity by matching the developer’s fidelity measure item to an OMLIT item, 
as closely as possible. For example, one BTL instructional practice involves engaging children in 
post-reading discussions of moderate or greater depth. The Read Aloud Profile (OMLIT-RAP) 
includes a quality rating as follows: 

1 = no post-reading extension or activities 

2 = discussion and/or activity that: 

relates to the story /book but does not extend its meaning or comprehension; 

BUT lasts less than 5 minutes. 

3 = discussion and/or activity that: 
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relates to the story /book but does not extend its meaning or comprehension; 

AND lasts at least 5 minutes. 

4 = discussion and/or activity that: 

reinforces comprehension of the story /book; 

BUT lasts less than 10 minutes. 

5 = discussion and/or activity that: 

extends the meaning of the text and reinforces comprehension of the story /book; 

AND lasts at least 10 minutes. 

Given these anchors, a score of '2' doesn't constitute moderate or greater depth, so the minimum to 
qualify for fidelity of implementation of BTL would be a score of '3'. 

Thus, we set the criterion that in order for a teacher to have demonstrated evidence of engaging 
children in post-reading discussions of moderate or greater depth, she needed to have obtained a 
rating of 3 or higher (out of 5) on a Read Aloud Profile (RAP) quality rating of the depth of post- 
reading activity. In another example involving an RSL! instructional practice, teachers are supposed 
to facilitate opportunities for students to practice reading and writing. In order to have demonstrated 
evidence of this practice, teachers had to have ratings above 2 (out of 5) for both the frequency and 
quality of reading and writing activities measured by the QUILL. 
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Exhibit A1 : 



Mapping Categories from BTL Fidelity Checklist to OMLIT Variables and Constructs for BTL Implementation Index 



Items from BTL Fidelity Checklist 


Items from BTL Implementation Index 


Reads & Discusses 






Comprehension while reading aloud & Comprehension in non-reading 
activities (* indicates activity in small and large groups) 


1 . BOW displayed prominently in the classroom (such as on an easel, easy 
to see) 


• Not measured 


2. Graphic organizers displayed and used in instruction 4-5 times per week 


• Not measured 


3. All/Most BTL reading support materials visible and accessible (Elephant, 
BOW posters, Alpha frieze, audio CDs, pupil books, book baskets). Visible & 
accessible: displayed at eye level of children, used by children. 


• Not measured 


4. BOW is read aloud daily or as often as possible 


• The teacher read a book aloud. 

• The teacher read aloud a BTL Book of the Week (BOW). 

• The teacher read a book connected to a class theme. 


5. Teacher meets with small groups to re-read the BOW. 


• The teacher read a book aloud — in small groups. 

• The teacher read aloud a BTL Book of the Week (BOW) — in small groups.. 

• The teacher read a book connected to a class theme — in small groups.. 


6. Teacher uses BTL Teacher Guide to provide comprehension and word study 
activities for small groups, determined by child reports, performance and 
observation data. 


• The teacher used one or more strategies promoting comprehension while 
reading aloud.* 

• The teacher made at least 1 connection between the book being read and 
children's experiences or class themes.* 




Vocabulary & Oral Language 


7. Teacher emphasizes vocabulary/ oral language 4-5 days during a typical 
week. 


• The teacher emphasized vocabulary in at least one read aloud.* 

• The quality of story-related vocabulary instruction during read alouds was 
moderate to high (e.g., one or more vocabulary word was discussed and a 
comprehension support was used). 

• The teacher provided one or more activities that supported vocabulary 
development outside of read-alouds (e.g., vocabulary knowledge was 
provided in the context of explanations, writing, songs, stories, rhymes, 
language games, discussions, shared reading, emergent writing, and/or 
child tagging or matching).* 

• The teacher provided at least one literacy activity aimed at developing 
children's oral communication and/or listening skills. 

• The teacher had students engage in language games, rhymes, songs, 
storytelling, or discussion in order to comprehend a text. If the activity was 
a discussion, it was rated moderately high or high (e.g., they included turn- 
taking by students and the teacher elaborated or asked students to 




Project Upgrade Implementation Study A-6 



Exhibit A1 : 



Mapping Categories from BTL Fidelity Checklist to OMLIT Variables and Constructs for BTL Implementation Index 



Items from BTL Fidelity Checklist 


Items from BTL Implementation Index 




elaborate on comments).* 

• The teacher used one or more instructional strategies that promoted higher 
order thinking during or after a read-aloud session (e.g., they asked book- 
related open-ended questions that required speculation, and/or expanded 
responses, thinking, or analyses).* 

• The teacher provided at least a few opportunities for oral language. 

• The average quality of the oral language activities provided by the teacher 
was moderate to high (e.g., the teacher provided integrated, higher quality 
oral language opportunities of varying types for most students individually 
or in small groups in which the teacher scaffolded or extended discussions 
with multiple turns that focused on non-management topics). 


8. Teacher emphasizes comprehension strategies/higher-order thinking skills 4- 
5 days during a typical week. 


• The average depth of post-read-aloud discussions or activities was 
moderate to “high” (e.g., they lasted at least 5 minutes and/or they 
extended or reinforced the comprehension of the book).* 

• The teacher provided one or more activities that supported the 
development of comprehension skills outside of read-alouds.* 


Writes 




1 . Child writing related to BOW and other 
independent writings, beyond names, are displayed. 


• Examples of children's writing were on display in the classroom (other than 
just names). 


2. Children write in journals daily 


• Children spent time in journal-writing activities. 


3. Teacher maintains writing folders to track children’s progress, and, s/he consull 
writing folders to make 

instructional decisions such as: which updates to implement 
for ISI, small group work, etc. 


• Not measured 


4. Teacher offers a wide array of daily writing opportunities, 
including authentic activities such as captioning and 
journaling, connected to BOW. 


• The teacher used at least one high quality activity aimed at developing 
knowledge of the functions and features of print (e.g., using authentic print 
materials, graphic organizers, or word webs). 

• The teacher provided a separate writing area and/or writing materials 
accessible to children. 

• The teacher provided one or more literacy activities that involved emergent 
writing, copying, or tracing. 

• Children were engaged in one or more writing activity (including emergent 
writing, copying, and tracing) that was connected to a class theme during 
the day. 

• The teacher provided two or more opportunities to engage in writing. 
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Mapping Categories from BTL Fidelity Checklist to OMLIT Variables and Constructs for BTL Implementation Index 



Items from BTL Fidelity Checklist 


items from BTL Implementation Index 




• The teacher provided three or more types of writing activities. 

• The teacher provided children with writing opportunities that were 
sometimes, often, or consistently of higher quality (e.g., emergent writing, 
captioning, dictating, writing names on work, book-making, and/or writing in 
journals. 

• The teacher provided writing opportunities that were sometimes or often 
integrated into activities with goals other than literacy. 


5. Teacher encourages children to begin to write using scribbles, drawings, 
copying letters and words, and some phonetic spelling. 


• The teacher sometimes, often, or consistently encouraged children to write 
on their own rather than insisting on conventional letter formation or 
spelling. 


6. Teacher initiates shared, interactive, and guided writing strategies to support th 
children as they work toward 

becoming independent writers, [teacher uses all three] 


• In at least some writing activities, the writing was done by the children 
themselves rather than by adults. 

• The teacher worked with most or all of the students in writing activities 
during the day. 

• Writing activities and opportunities were sometimes, often, or consistently 
conducted with individuals or small groups. 


7. Teacher always uses graphic organizers in coordination 
with the BOW, as a springboard to move children 
from oral discussion to writing. 


• Not measured. 


Interactive Software Instruction (ISI) 




1 . The average child used a computer for the amount of time recommended by 
BTL, or longer (pre-K: 8-12). 


• The average child used a computer for the amount of time recommended 
by BTL, or longer (pre-K: 8-12). 


2. Teacher uses BTL ISI as an instructional tool, AND regularly reviews 
updates. 


• Not measured. 


3. Most children progressing appropriately in Explore Words objectives, AND 
there is evidence of the Explore Words curriculum in other classroom activities. 


• Not measured. 


4. Teacher uses reports analytically with assessments to place students, 
differentiate instruction, and confirm observations. 


• Not measured. 


5. Teacher consistently models effective ISI use and uses reports to identify 
areas where modeling is needed. 


• Not measured. 


Classroom Culture and Management Routines 




1 . Teacher has established a consistent system and schedule and moves the 
class between activities smoothly and efficiently. 


• The average child spent less than 33% of the time in transitions, routines, 
and management or being uninvolved in activities. 

• 


2. Children independently manage computer turntaking using established 


• Not measured. 
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Mapping Categories from BTL Fidelity Checklist to OMLIT Variables and Constructs for BTL Implementation Index 



Items from BTL Fidelity Checklist 


Items from BTL Implementation Index 


system . 




3. Teacher has well-structured room and leads discussions in whole or small 
groups, AND children read, reflect on, and work in whole and small groups and 
in literacy centers. 


• The classroom was well-organized and had the space and layout to afford 
children a variety of independent activity choices, including learning centers 
(e.g. it had at least one distinct activity center, some materials that were 
marked, sorted, and stored; the layout allowed for at least two group sizes, 
some independent choices for students, sufficient space, adequate light, 
and/or no odors). 


4. Children are provided a variety of learning experiences and choices daily - 
independent, whole and small group instruction, learning centers. 


• The average child spent at least 50% of their time engaged in educationally 
“high value” activities (including: reading, alphabet, oral language, sounds, 
writing, science/nature, math concepts, dramatic play, creative play, block 
play, and fine motor play). 

• The average child spent at least 20% of their time in groups of 5 or fewer 
(not including meals, routines, transitions, or management). 
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Items on RSL! Fidelity Checklist 


Construct Measured on RSL! Implementation Index 


1. Interaction 




1. The tone and classroom conversations are positive, teacher shows respect 
for children and listens attentively, encouraging them to learn likewise. 


Caregiver speaks warmly and listens attentively to children 


2. Teacher encourages children to express their feelings, experiences, and 
perspectives with intentional efforts to expand their spoken vocabulary 


Not Measured 


3. Evidence of full-group activities, where each child is encouraged to 
participate 


Evidence of full-group activities. 


II. Environment 




1. In the classroom: Furnishings and materials appropriately sized, in good 
repair, clearly organized, appealing, accessible and not crowded 


In the classroom, materials are organized, and space in the classroom is 
sufficient size for the number of children 


2. Learning areas provide opportunities for children to engage independently 
in activities 


The classroom layout allows children to choose materials and participate in 
activities independently 


3. An inviting and orderly area is set aside for book reading, made comfortable 
by soft materials such as pillows and cushions 


There is a distinct reading area, with books to choose from, where children can 
sit comfortably to read, with soft furnishings, including a rug and pillows, 
cushions or couches. 


4. Books promote cultural diversity and there are materials in other languages 
that represent present children’s culture of origin 


There are books accessible to children in languages other than English, and 
either books show diverse groups of people or toys/materials represent varied 
cultures, ethnic groups, or types of people. 


5. There are word cards, with names or familiar words intended to support 
children’s writing 


There are tovs and materials accessible to children that include words/letters 
(e.g., toys: puzzles, blocks, board games, card games; materials: magnetic 
letters, letter stamps, letter cards, word cards). 

If include cloc_col32 - 

(and there are tools accessible to children to help them practice writing 
words/letters (e.g., stencils, templates, tracing sheets, worksheets). 


6. There is variety of paper and writing tools available (chalkboard, pencils, 
crayons, etc) 


There is a variety of paper types and/or a variety of writing tools accessible to 
children. 


7. Variety of teacher’s dictation (teacher-led activity, from child’s work) and 
children's writing (journals, sign in, etc) on display 


Examples of children’s own writing, other than just the child's own name on 
work/artwork, are on display. 


III. RSL! Curriculum 




1. RSL! Materials are set up throughout the classroom 


Not Measured 


2. The teacher has implemented a system that ensures that children 
appropriately use the RSL! technology components during learning center time 
and have learned to take care of it (please document teachers system on 
comment sheet) 


Not Measured 
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Mapping Items of RSL! From Original Fidelity Instrument onto OMLIT Measures for RSL! Implementation Index 



Items on RSL! Fidelity Checklist 


Construct Measured on RSL! Implementation Index 


3. There is evidence in the classroom of the use of RSL! Materials 


Not Measured 


4. The teacher follows the daily schedule 


Not Measured 


5. Opportunities for extended uses of language, (activities, materials and 
displays) are coordinated with RSL! goals in each unit and its current theme 


Classroom has a theme based on a rich concept, and theme is evident in 
classroom materials or displays. 


6. The teacher follows a lesson plan from the “Teacher’s Manual” or the one 
provided by the project coordinator 


Not Measured 


7. Teacher encourages the use of the LeapMat to review the letters learned in 
the current lesson, and follows suggested activities in the “Teacher's Manual” 


The LeapMat is used as a literacy resource in the classroom. 


8. Teachers integrates the LeapPad, the correspondent book, and activities 
(“Teacher’s Manual”) with the current lesson 


The LeapPad is used as a literacy resource in the classroom. 


9. Teachers encourage the use of the LeapDesk , and models its use to 
review letters and numbers as suggested in the Manual 


The LeapDesk is used as a literacy resource in the classroom. 


1 0. Teacher reads the suggested book for the lesson (Read-Aloud Library) , 
and implements suggested activities in the “Teacher's Resource Guide” 


Teacher reads a book related to the class theme, and organizes post-reading 
activities related to the book or other activities related to the theme. 


1 1 . The teacher integrates “ Little Ears” CD or tape to the lesson (note in 
comments section if music is simply played vs. actively integrated) 


Not Measured 


12. The teacher uses the Big Books and Read Aloud Library for full-group and 
small group activities according to the lesson 


Teacher or children read a big book related to the class theme and engage in 
an activity related to the book/theme. 


13. Teaching strategies support reading comprehension, questioning, 
predicting, summarizing, problem-solving, and reflecting on learning 


During reading, teacher uses open-ended questions that support reading 
comprehension by requiring prediction, thinking or analysis, by expanding 
responses, or by relating the book to children’s experiences. 


14. Adults engage in one-to-one or small group activities using RSL! materials 


Adults use RSL! materials in activities with children, one-on-one or in small 
groups. 


15. Adults facilitate opportunities and model for the children to practice their 
reading and writing skills as learned in the lessons 


Adults provide a moderate number of opportunities for children to engage in 
language and literacy activities and a moderate number of opportunities for 
children to engage in writing activities, and these reading and writing activities 
are above minimal quality. 


16. Children use RSL! Materials on their own or with a friend by own initiative 


Children use RSL! materials on their own or with a friend, 
(no info about doing so by own initiative) 


17. RSL! Poem and song posters are posted as for correspondent lesson and 
used along with the “Little Ears” CD 


Not Measured 


18. Teachers uses the Flip Book for discussions to increase of words used 
and understood, and practice more complex sentences 


Not Measured 
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Items on RSL! Fidelity Checklist 


Construct Measured on RSL! Implementation Index 


19. Puppets are used as learning aids and as suggested by lesson 


Not Measured 


20. Teacher demonstrates mastery of RSL! Materials and its integration with 
the current lesson 


Not Measured 


IV. Assessment 




21 . Information gained from assessment is used to plan or vary practices 
accordingly to ensure children’s individual growth and achievement 


Effective use of assessment (to direct instruction) 
Not Measured 


22. Class assessment report is completed by the end of each unit, to be aware 
and monitor children’s progress according to the unit’s objectives 


Effective use of assessment (to monitor children’s progress) 
Not Measured 


V. Family Involvement 




23. There is evidence in the classroom of attempts to learn and use 
information from children’s homes 


Family involvement 
Not Measured 


24. There are regular interactions between home and school to support 
children’s learning, by guiding them through RSL! materials and assignments, 
and to provide information about children’s progress 


Family involvement 
Not Measured 


25. Teacher has an effective check-in and out system in place for parents to 
use the RSL! materials at home 


Family involvement 
Not Measured 
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Mapping Items of From B.E.L.L. Fidelity Checklist onto OMLIT Variables for B.E.L.L. Implementation Index 



Items in B.E.L.L. Fidelity Checklist 


Possible OMLIT Variables 


Comments 


Phonological and Print Awareness Component: 
Oral Language Development 






1 . Teacher discusses related concepts & vocabulary 


CLIP: Discussed vocabulary. 


Combination of OMLIT codes not found. — not 
coded, not measured 


2. Teacher asks questions related to rhyme 
(meaning) 


CLIP: Discussed comprehension of text - coded 
with “rhyme chart” 


Combination of OMLIT codes not found. — not 
coded, not measured 


3. Teacher & students recite or sing rhyme 


CLIP: Presented oral communication or listening 
skills or sound (no print) or led a singing activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


4. Teacher & students dramatize rhyme 


Not measured 


Not measured 


Phonological Awareness: Word Awareness 






5. Students push blocks for each word 


Not measured 


Not measured 


6. students clap for each word 


Not measured 


Not measured 


Phonological Awareness: Rhyming Words 






7. use rhyming objects or pictures 


CLIP: Sounds-focused activity with objects or 
pictures 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


8. make a rhyming picture web 


Not measured 


Not measured 


9. play “thumbs-up/thumbs-down” game 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


Phonological Awareness: Compound words / 
Syllables 






10. Use 2-part pictures or puzzles 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


1 1 . feel syllables with hand under chin 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


12. use connecting links or blocks to show 
awareness of syllables 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


13. use fists to “break it and make it” 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


Phonological Awareness: Alliteration 






14. use beginning sound objects/pictures 


CLIP: Sounds-focused activity 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


15. make a beginning sound picture web 


Not measured 


Not measured 


Print Awareness & Alphabet Knowledge 






16. demonstrate one-to-one correspondence 


CLIP: Activity focused on functions/features of print 


B.E.L.L. item more specific than OMLIT — not 



Project Upgrade Implementation Study A-13 



Exhibit A3: 
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Items in B.E.L.L. Fidelity Checklist 


Possible OMLIT Variables 


Comments 






measuring B.E.L.L. 


17. demonstrate return sweep 


CLIP: Activity focused on functions/features of print 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


18. demonstrate top to bottom & left to right 


CLIP: Activity focused on functions/features of print 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


19. point to beginning letters on charts 


CLIP: Activity focused on functions/features of print 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


20. sing the ABC song with Alphabet Book 


Singing with “ABC song” and alphabet book 




Additional Classroom Activities 






21. Use letters in children’s names 


CLOC: Presence of children’s names on cards 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


22. use magnetic or plastic letters 


CLOC: Presence of letter-related materials 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


23. use alphabet books, puzzles, games 


CLOC: Presence of letter-related materials 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


24. use labels, charts, signs, messages 


CLOC: Presence of environmental print 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


Miscellaneous 






25. Rhyme chart used 


Not measured 


Not measured 


Oral Language & Shared Reading Component: 
Pre-reading (develop background knowledge / 
schema) 






26. discussion: concepts/vocabulary 


RAP: vocabulary focus in pre-reading 




27. front cover: predictions 


RAP: predictions in pre-reading 




28. tell a personal story 


RAP: tell story before reading 




29. picture walk 


RAP: picture walk 




30. children make up their own story 


Not measured 


Not measured 


Oral Language & Shared Reading Component: 
Read TO children (model & enjoy story) 






31. use appropriate intonation 


RAP: reads with expression 




32. focus on picture cues 


RAP: calls attention to pictures during reading 




33. comprehension questions 


RAP: asks comprehension questions during reading 




34. children retell story using pictures 


RAP: post-reading retelling 


Too specific (OMLIT doesn’t distinguish teacher’s 
from children’s retelling) — not measured 
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Items in B.E.L.L. Fidelity Checklist 


Possible OMLIT Variables 


Comments 


Oral Language & Shared Reading Component: 
Re-read WITH children (discuss & interact) 






35. invite children to “choral read” 


RAP: children join in reading 




36. reread / echo read predictable lines 


RAP: children join in reading 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


37. discuss story (characters, plot, setting) 


RAP: post-reading discussion 


B.E.L.L. item more specific than OMLIT — not 
measuring B.E.L.L. 


38. ask open-ended questions 


RAP: post-reading discussion — quality 




39. relate story to children’s life experiences 


RAP: post-reading ask question/relate story to 
children’s life experiences 




Oral Language & Shared Reading Component: 
React 






40. [use] puppets 


RAP: text support = puppets 




41 . illustrate class book 


Not measured 


Not measured 


42. dramatize 


RAP: children act out story 




43. hands-on activity 


Not measured 


Too general — Not measured 


44. structure writing 


RAP: post-reading activity 


Too specific — not measured 


Oral Language & Shared Reading Component: 
Book Awareness 






45. show cover 


RAP: teacher shows/comments on cover, points to 
title, points to author/illustrator 


Common practices — not specific to B.E.L.L. 


46. point to the title 


Same as for item 45 


Common practices — not specific to B.E.L.L. 


47. point to the author 


Same as for item 45 


Common practices — not specific to B.E.L.L. 


48. point to the illustrator 


Same as for item 45 


Common practices — not specific to B.E.L.L. 


49. pictures tell a story 


RAP: teacher does “picture walk” 


Common practices — not specific to B.E.L.L. 


50. teacher reads the print 


RAP: teacher reads a book aloud 


Common practices — not specific to B.E.L.L. 
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Observation Measures for Language and Literacy Instruction (OMLIT) 1 

The OMLIT (Observation Measures of Language and Literacy Instruction) was a new battery developed 
for the national study of the Even Start Family Literacy Program being conducted by the U.S. Department 
of Education. The CLIO 2 study was also an experimental test of early childhood language and literacy 
curricula, and, as with the Miami study, CLIO needed measures of classroom process that would be 
sensitive to the interventions. The CLIO study also reviewed available measures, including the Early 
Language and Literacy Observation Tool (ELLCO) 3 and the ECERS-R, and determined that new 
measures would have to be developed if measuring effects on classroom process was a priority. The 
Department of Education supported the development of the OMLIT battery, with the charge that the 
measure would be closely linked to the most up-to-date research on instructional practices shown to 
predict children’s reading and other academic outcomes in school. Given the more than adequate 
reliability of the OMLIT battery, its clear link to all of the critical classroom outcomes in the study, and 
its track record in large-scale applied research, we selected the OMLIT for the Miami study. Although we 
considered administering the ECERS-R together with the OMLIT, for puiposes of comparison with other 
early childhood studies, we judged that the two measures would have to be administered in separate visits 
to classrooms (i.e., observers could not reliably code both the OMLIT and the ECERS-R simultaneously). 
The cost of the additional training and doubling the visits to classrooms was determined to be prohibitive, 
especially in light of the limited usefulness of the ECERS-R for measuring treatment-control differences 
(versus allowing us to characterize the quality of the child care centers in the Miami sample compared 
with other samples). 

The complete battery of observation measures used includes five instruments from the Observation 
Measures for Language and Literacy Instruction (OMLIT; Goodson, Layzer, Smith, Rimdzius, 2004) 
battery and the Arnett Caregiver Rating Scale (Arnett, 1989). 4 

The Snapshot of Classroom Activities (OMLIT-Snapshot) 

The OMLIT-Snapshot is a description of classroom activities and groupings, integration of literacy in 
other activities, and language in the classroom. It has two sections. The Environment section describes the 
number of children and adults present, as well as the type of adult (staff, parents, etc.). The Activities 
section describes activities that are taking place. For each activity, the observer records the number of 
children and adults in that activity, whether any adult or child is talking, and whether they arc speaking 
English or another language, and whether literacy materials are used (text, writing, letters, singing). 



1 The OMLIT (Goodson, Layzer, Smith, Rimdzius, 2004) is an unpublished battery of observation instruments 
for use in measuring the quality of early language and literacy instruction in ECE classrooms. 

The study is named CLIO, for Classroom Literacy Intervention and Outcomes study. 

3 Like the ECERS-R, the version of the ELLCO available at the time of the study focused primarily on 
environmental supports for literacy (classroom materials, activities) with little attention to teacher-child 
interactions which we believed to be critical elements of support for language and literacy development. 

4 The Arnett measures the teacher’s affective behavior and disciplinary style. 
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The Read Aloud Profile (OMLIT-RAP) 



The OMLIT-RAP is a description of staff behavior when reading aloud to children (in CLIO, the RAP 
was completed when an adult was reading to at least two children). The RAP records adult behavior 
during the read-aloud session in four categories: (a) pre-reading (set-up) behavior, (b) behavior while 
reading the book, (c) post-reading behavior, and (d) the language the adult uses when talking to children 
during the read aloud. The RAP records characteristics of the adult, the children, and the book itself in 
three categories: (a) role of the adult involved in the read-aloud (e.g., teacher, aide), (b) characteristics of 
the book being read, and (c) number of children involved in the read-aloud. The RAP also includes five 
quality indicators which su mm arize particular aspects of the read-aloud: (a) the degree to which the adult 
introduces and contextualizes new vocabulary to support children’s learning, (b) the depth of the 
discussion related to the story that the adult facilitates with the children before, during, and after the read- 
aloud, (c) the extent to which the adult uses open-ended questions that invite children to engage in 
prediction, imagination, and/or rich description, (d) the depth of children’s engagement with the read- 
aloud activity, and (e) the quality of any post-reading book-related activities that the adult organizes 
(beyond oral discussion). 

The Classroom Literacy Opportunities Checklist (OMLIT-CLOC) 

The OMLIT-CLOC is an inventory of classroom literacy resources. It provides an overall rating of the 
extent to which a classroom is a literacy-rich environment and delineates eight aspects of the literacy 
environment: (a) physical layout of the classroom, (b) the text or print environment, (c) books and reading 
or listening areas, (d) writing resources, (e) literacy-related materials and toys, (f) cultural diversity in 
literacy materials, (g) literacy integrated in classroom areas or learning centers, and (h) the richness and 
integration of a curriculum theme. 

The Classroom Literacy Instruction Profile (OMLIT-CLIP) 

The OMLIT-CLIP involves a two-stage coding protocol in which the observer first determines if any 
classroom staff member is involved in a literacy activity and, if so, the observer codes seven 
characteristics of the literacy activity: the type of activity, the literacy knowledge being afforded to the 
children, the adult’s level and type of participation in the activity, any text support, languages spoken by 
staff and children, and the number of children involved. If the literacy activity involves adult-child 
discussion, the quality of this discussion is rated on three characteristics — the cognitive challenge in the 
discussion, the extensiveness of the discussion, the level of abstraction of the discussion. 

The Quality of Language and Literacy Instruction (OMLIT-QUILL) 

The OMLIT-QUILL is an overall evaluation of the quantity and quality of the instructional practices that 
build children’s print awareness and oral language skills, expose children to a rich and varied vocabulary, 
and build children’s phonological awareness. These practices are predictors of better reading outcomes 
for children once they are in school; this is particularly true of those at risk for reading difficulties 
(Dickinson and Tabors, 2001; Lonigan, Burgess, and Anthony, 2000; NICHD, 2000; Snow, Burns, and 
Griffin, 1998; Whitehurst and Lonigan, 1998). In addition, the QUILL evaluates instructional practices 
with English language learners. 

More information about the OMLIT can also be found in the earlier report on Project Upgrade: 

Evaluation of Child Care Subsidy Strategies: Findings from an Experimental Test of Three 
Language/Literacy Interventions in Child Care Centers in Miami-Dade County, Final Report (January 
2009). 
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